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PREFACE 


This  volume  (Volume  I.  Planning  and  Management)  is  one  part  of  a 
three-volume  Handbook  produced  for  the  U.S.  Air  Force  Human  Resources 
Lahoratory/Operations  Training  Division  (AFHRL/OT).  The  Handbook  is 
entitled,  "Handbook  for  Operational  Test  and  Evaluation  ( OT & E )  of  the 
Training  Utility  of  Air  Force  Aircrew  Training  Devices."  This  effort 
has  been  accomplished  by  the  Seville  Research  Corporation  under  Con¬ 
tract  No.  F33615-78-C-0063.  Dr.  Thomas  H.  Gray  served  as  the  Air 
Force  Laboratory  Contract  Monitor  (AFLCM)  on  the  project.  For 
Seville,  Dr.  William  H.  Hagin  was  Project  Director,  and  Dr.  Wallace  W. 
Prophet  was  Program  Manager. 

The  three  volumes  which  comprise  the  total  Handbook  are  intended 
to  provide  guidelines  and  procedures  appropriate  for  use  of  Air  Force 
ATD  OT&E  test  team  personnel  in  planning,  conducting,  and  reporting  the 
results  of  aircrew  training  device  OT&E  efforts.  The  three  Handbook 
volumes  are: 

Volume  I.  Planning  and  Management 

Volume  II.  Operational  Effectiveness  Evaluation 

Volume  III.  Operational  Suitability  Evaluation 

It  is  important  that  the  reader  understand  that  this  Handbook  was 
prepared  to  serve  as  a  supplement  to  AFM  55-43,  "Management  of  Opera¬ 
tional  Test  and  Evaluation"  by  providing  those  specific  additional 
evaluation  concepts  and  techniques  necessary  for  ATD  test  and 
eval uation. 
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CHAPTER  1 


HANDBOOK  OVERVIEW 


\ 

INTRODUCTION 

The  U.S.  Air  Force  has  utilized  ground-based  aircrew  training 
devices  (ATDs)  to  support  inflight  aircrew  training  programs  for  many 
years.  The  devices  used  have  included  cockpit  famil iarization  and 
procedures  trainers,  part- task  trainers,  instrument  flight  simulators, 
mission  trainers,  operational  flight  trainers,  and  complex  weapon 
system  trainers.  Until  relatively  recently,  ATDs  were  viewed  as  "nice 
to  have"  additions  to  the  use  of  aircraft  for  training,  but  not  as 
critical  elements  in  training  programs.  The  contribution  to  aircrew 
training  effectiveness  of  famil  iarization  and  procedures  trainers, 
part-task  trainers,  and  even  instrument  flight  simulators  was  essen¬ 
tially  presumed.  Furthermore,  there  were  no  formal  requirements  to 
justify  ATD  acquisition  on  the  basis  of  device  versus  aircraft  cost- 
benefit  trades. 

Importance  of  ATD  T raining 

The  1974  oil  embargo-caused  fuel  shortage  is  widely  recognized 
within  the  Air  Force  as  having  been  a  "trigger"  for  the  generation  of 
greater  concern  about  the  real  training  value  of  the  full  array  of 
aircrew  training  devices.  At  all  echelons  of  Air  Force  management, 
there  was  an  increased  awareness  of  the  need  to  define,  much  more  pre¬ 
cisely  than  had  ever  before  been  required,  the  role  of  ATDs  in  all 
aspects  of  Air  Force  aircrew  training--from  the  undergraduate  level  to 
that  of  the  combat  mission  leader. 

Concern  for  ATD  training  effectiveness  is  particularly  acute 
today  in  view  of  the  almost  prohibitive  cost  of  aviation  fuel-- 
especially  when  present  fuel  costs  are  projected  toward  the  future. 
The  role  of  ATDs  as  an  essential,  if  not  critical,  component  of  Air 
Force  aircrew  training  is  unquestioned .  As  a  result  of  significant 
advances  in  both  flight  simulation  and  aircrew  training  technologies, 
the  potential  contribution  of  ATDs  toward  improved  quality  of  aircrew 
training,  and  to  substantial  cost  avoidance  in  the  conduct  of  that 
training,  has  become  increasingly  recognized  and  ATDs  have  become  a 
major  component  of  nearly  all  USAF  aircrew  training  programs.  In 
fact,  there  are  some  aircrew  training  programs  in  which  the  trainee 
spends  almost  as  much  time  in  ATDs  as  he  does  in  the  aircraft  itself. 

From  the  perspective  of  enhancing  combat  readiness  training,  the 
advantages  of  modern  ATDs  are  rema-kable.  State-of-the-art  ATDs 
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enable  aircrews  to  practice  combat  flying  skills  without  regard  to 
weather  conditions,  aircraft  downtime,  insufficient  weapons  ranges, 
and  limited  live  ordnance  (missiles  and  ammunition).  They  allow 
trainees  to  practice  with  simulated  friendly  and  threat  aircraft,  sub¬ 
marines  and  surface  forces,  and  electronic  countermeasures .  These 
capabilities  permit  aircrews  to  maintain  enhanced  levels  of  combat 
readiness  that  would  be  otherwise  unattainable.  Of  course,  ATDs  do 
not  eliminate  the  necessity  for  flying  aircraft.  Aircraft  must  always 
be  flown  to  polish  combat  skills,  to  exercise  critical  maintenance  and 
logistics  support  systems,  and  to  validate  the  airborne  effectiveness 
of  those  techniques  and  procedures  which  were  learned  in  ATOs. 

Need  for  Quality  ATO  Evaluation  Data 


The  present  role  of  ATDs  in  support  of  Air  Force  aircrew  training 
makes  it  more  important  than  ever  before  that  the  Air  Force  have  a 
means  of  assuring  that  ATD  training  is  effective.  As  increased  com¬ 
mitments  are  made  to  the  exploitation  of  ATDs  in  enhancing  aircrew 
training,  and  as  training  resource  costs  (personnel,  fuel,  munitions, 
etc.)  escalate,  more  and  more  aircrew  training  necessarily  will  be 
conducted  in  ATDs.  It  is  important,  therefore,  that  the  Air  Force 
have  a  suitable  data  base  which  demonstrates  convincingly  that  these 
devices  do,  in  fact,  meet  operational  training  needs. 

The  ATD  training  effectiveness  literature,  upon  which  the 
presently  available  data  base  rests,  consists  mainly  of  numerous 
research-oriented  ATD  study  reports,  although  there  is  a  growing 
number  of  actual  ATD  07SE  reports.  This  developing  data  base  will  be 
used  to  support  many  critical  ATD  acquisition  and  utilization  deci¬ 
sions.  It  is  important,  therefore,  that  future  ATD  OT&Es  be  conducted 
with  the  maximum  precision  that  circumstances  will  allow  so  as  to  pro¬ 
duce  the  highest  quality  ATD  evaluation  data  possible. 

Handbook  Purpose 

AFM  55-43,  "Management  of  Operational  Test  and  Evaluation" 
explains  the  Air  Force  operational  test  and  evaluation  program  (OT&E) 
as  it  exists  to  implement  the  general  testing  and  evaluation  policies 
outlined  in  AFR  80-14,  "Operational  Test  and  Evaluation."  AFM  55-43 
is  directed  toward  meeting  the  needs  of  both  the  OT&E  manager  and  the 
"doers."  Volume  I  of  that  manual  provides  background  information  and 
general  guidelines  for  planning,  managing,  conducting,  and  reporting 
OT&Es.  Volume  II  of  the  manual  contains  highly  specific  procedures 
and  techniques  for  accomplishing  the  full  range  of  OT&E  activities. 
This  material  includes,  for  example,  specification  of  detailed  test 
procedures,  standard  formats  for  plans  and  reports,  checklists,  etc. 

The  guidance  provided  in  AFM  55-43  is  intended  to  apply  to  all 
Air  Force  0T£E--incl uding  the  test  and  evaluation  of  aircrew  training 
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devices.  Experience  has  shown,  however,  that  there  are  specific 
aspects  of  ATD  testing  for  which  substantial  supplementary  guidance  is 
needed.  It  is  the  purpose  of  this  Handbook  to  provide  that  supplemen¬ 
tary,  ATD-speci f ic,  information  and  guidance.  Although  it  is  intended 
primarily  for  use  by  typical  new  or  novice  test  personnel  with  subject 
matter  expertise  (e.g.,  a  qualified  pilot),  but  who  may  have  little  or 
no  previous  OT&E  experience,  the  Handbook  should  also  be  helpful  to 
more  experienced  OT&E  personnel. 

User  Guidance 


Since  this  Handbook  was  prepared  to  serve  as  a  supplement  to  AFM 
55-43,  "Management  of  Operational  Test  and  Evaluation,"  being 
thoroughly  familiar  with  the  content  of  that  manual  is  a  necessary 
prerequisite  to  effective  use  of  the  Handbook.  Users,  therefore,  are 
urged  to  "operationalize"  this  Handbook  as  an  ATD  OT&E-oriented  annex 
to  AFM  55-43:  Read  AFM  55-43  first  for  general  OT&E  guidance  and 
direction  and  read  (as  appropriate)  the  volumes  of  this  Handbook  for 
ATD-specific  instructions. 


HANDBOOK  ORGANIZATION  AND  CONTENT 

There  are  three  functions  with  which  the  ATD  OT&E  test  director 
must  be  concerned  and  for  which  Handbook  coverage  is  deemed  appro¬ 
priate.  The  first  of  these  functions  deals  with  overall  ATD  OT&E 
planning  and  management;  the  second  and  third  functions  are  concerned 
with  evaluating  the  actual  operational  effectiveness  and  suitability 
of  a  given  device.  The  Handbook  has  been  organized  in  alignment  with 
these  specific  functions  into  three  volumes: 

Volume  I:  Planning  and  Management 

Volume  II:  Operational  Effectiveness  Evaluation 

Volume  III:  Operational  Suitability  Evaluation 

Provided  below  are  brief  summaries  of  the  contents  of  each  of 
these  three  Handbook  volumes. 

V ol uine  I :  Planning  and  Management 

Volume  I  of  the  Handbook  serves  three  functions:  It  first  pro¬ 
vides  an  overview  of  the  content  of  the  entire  Handbook.  This  over¬ 
view  serves  somewhat  as  an  "Executive  Summary"  for  the  reader  who  may 
only  be  interested  in  the  big  picture  of  the  Handbook's  content. 
Second,  it  is  concerned  wi th  describing  both  general  and  specific  ATD 
OT&E  planning  and  management  considerations  and  links  those  events 
which  occur  early  in  the  ATD  acquisition  process  to  later  ATD  OT&E 
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planning  and  management  activities.  It  defines  the  various  evaluation 
concepts  that  are  germane  to  understanding  AID  OUt,  and  it  describes 
the  two  major  ATD  OT&E  activities:  -  Initial/Qual i fication  UT&E  and 
follow-on  OT&E . ( I OT&E/00T&E  and  FOTM  ) .  Major  emphasis  in  this  regard 
is  directed  toward  a  detailed  "roadmapping"  of  the  procedural  events 
and  milestones  with  which  ATD  OT&E  test  director  personnel  will  be 
concerned  in  managing  either  IOT&t/QOT&Es  or  FOT&ts.  Identified  also 
are  the  critical  personnel  and  materiel  resources  required  for  meeting 
the  more  important  of  these  events.  The  third  area  addressed  in 
Volume  I  concerns  matters  of  ATD  value  and  worth  to  the  Air  Force. 
The  acquisi tion  and  life  cycle  costs  associated  with  modern  ATDs  make 
such  concerns  important. 

C  hap te  r  1  ^  Introduction.  This  first  chapter  consists  of  brief 
overviews  of  the  content  of  the  three  Handbook  volumes.  These  over¬ 
views  are  intentional  ly  brief  so  that  the  reader  may  obtain  a  rela¬ 
tively  quick,  but  inclusive,  understandi ng  of  the  scope  of  the  entire 
Handbook.  Chapter  1  of  this  volume  thus  is,  in  effect,  an  Executive 
Summary  of  the  total  Handbook. 

Chd£ter  2_:  Understanding  ATD_  Acqui si U on  and  Test.  Chapter  2  of 
Volume  1  provides  a  general  background  discussion  of  the  ATD  acquisi¬ 
tion  and  test  process.  The  scope  of  this  discussion  ranges  from  the 
initial  Statement  of  tieed  (SOU),  through  the  acquisition  and  manufac¬ 
turing  stages,  to  the  device  test  and  evaluation  (TsE)  phase.  The 
coverage  of  T&E  presented  is  intended  to  give  tne  reader  a  basic 
understanding  of  the  major  types  of  Tm,  and  to  develop  an  awareness 
of  the  T&E  roles  and  responsibilities  assigned  the  various  Air  Force 
organi zations  involved  in  the  T&E  process. 

Chapter  3:  P  i  arLrl1i’1  and  Management  of  A'b  CT4£.  Chapter  5  spe¬ 
cifically  addresses  ATD  OT&E  planning  >r.d  ana  jement.  It  deals  with 
the  planning  and  management  of  both  Initial  and  Qual i fication 

Operational  Test  and  Evaluation  ( IOT&E/cO i &E )  and  Follow-on  Opera¬ 
tional  Test  and  Fvaluation  (FOT&E).  The  intent  of  this  chapter  is  to 
provide  the  ATD  UT test  team  an  understandi ng  of  the  basic  structure 
and  critical  milestones  for  the  planning,  conducting,  and  reporting  of 
an  AID  IOT&E/QOT&E  or  EUT&E. 

Chapter  4 :  ATD  l)T&E  Fvent  Flow  and  Resource  Requirements.  It  is 
important  that  the  test  di roc  tor/ team  have  a  clear  conceptual i nation 
of  the  "temporal"  flow  of  ATD  OT.VE  events,  tor  this  reason.  Chapter  4 
presents  an  hypothetical  (but  nonetheless  representative)  " time- 1 i ni ng" 
of  the  ATD  acquisition  and  OT&E  process.  In  that  time-line,  a  dis¬ 
tinction  is  made  between  events  that  occur  while  the  ATD  is  ;  a- plant 
or  on-site  and  between  pro-  and  post- acceptance  testing  01 &E  activity. 
Of  significance  in  that  presentation  also  is  the  identification  of 
critical  Of&E  supporting  manpower  and  nn'm  id  resource  requirements. 


Volume  I  concludes  with  a  Glossary  of  Terms  (Appendix  A)  and  an 
Annotated  Bibliography  of  selected  Air  Force  documents  and  research 
reports  (Appendix  B) . 

Volume  II:  Operational  Effectiveness  Evaluation 

Volume  II  treats  the  assessment  of  ATD  training  during  OT&E. 
Because  of  its  direct  importance  to  ATD  training  effectiveness  eval¬ 
uation,  and  because  AF M  55-43  provides  little  ATD-specific  effec¬ 
tiveness  evaluation  information,  this  volume  of  the  Handbook  is 
somewhat  exhaustive  in  its  treatment  of  the  effectiveness  evaluation 
subject  matter.  Volume  II  has  seven  chapters.  The  first  four  chap¬ 
ters  basically  comprise  a  tutorial  on  the  fundamental s  of  ATD  utiliza¬ 
tion,  evaluation  concepts,  and  evaluation  approaches.  The  next  two 
chapters  address  specific  training  evaluation  techniques  as  they  per¬ 
tain  to  direct  evaluation  of  an  ATD's  training  effectiveness.  The 
final  chapter,  in  recognition  of  the  fact  that  ultimate  ATD  effective¬ 
ness  is  also  very  much  a  function  of  its  instructor/operator  station 
(IDS)  design,  provides  highly  specific  guidance  for  IOS  evaluations. 

Chapter  1:  Introduction.  Chapter  1,  as  its  title  suggests,  is 
mainly  an  introduction.  It  defines  the  intent,  scope,  and  content  of 
Volume  II  and  "roadmaps"  its  organization.  It  also  provides  a  very 
brief  overview  of  the  content  of  each  of  the  remaining  chapters, 
Chapters  2  through  7,  inclusive. 

Chapter  2:  Aircrew  Training.  This  chapter  discusses  the  nature 
of  aircrew  training.  An  important  distinction  is  made  in  this  chapter 
between  initial  aircrew  training,  as  contrasted  with  continuation 
training  or  the  maintenance  of  already  acquired  skills.  The  import  of 
that  distinction  for  ATD  utilization  and  planning  is  emphasized 
strongly  in  this  chapter.  It  is  stressed  that  the  test  team  must  be 
fully  cognizant  that  the  effectiveness  of  ATDs  may  not  be  the  same  for 
both  categories  of  training  and  that  ATD  OT&Es  need  to  be  conducted  in 
the  specific  context  of  their  intended  appl ication--initial  and/or 
continuation  training. 

C hapter  3:  ATD  Effectiveness:  Definition  and  Evaluation  Methods . 
Chapter  3  of  Volume  II  continues  the  thrust  of  the  distinction  in 
Chapter  2  between  initial  skill  acquisition  and  subsequent  skill  main¬ 
tenance,  from  the  perspective  of  AID  effectiveness.  It  also  provides 
several  definitions  of  ATD  effectiveness,  including  an  "official"  Air 
Force  definition.  Having  thus  communicated  an  awareness  of  the 
meaning  of  ATD  effectiveness,  the  chapter  then  summarily  reviews  a 
number  of  evaluation  models  that  have  been  used  within  either  the 
research  or  the  training  applications  community  to  evaluate  ATD  effec¬ 
tiveness.  Chapter  3  also  develops  a  distinction  between  those  eval¬ 
uation  models  that  employ  analyses  or  judgments  obtained  from  subject 
matter  experts  (analytic  models)  and  those  evaluation  models  that 


involve  actual  training  (demonstrations  models).  This  distinction  is 
useful  to  the  test  team  in  the  context  of  IOT&E/QOT&E  and  FOT&E. 


This  review  is  not  intended  to  endorse  or  reject  any  one  of  the 
models  per  se,  but  is  provided  for  general  information  purposes. 
Recommendations  for  specific  evaluation  model  applications  are  pro¬ 
vided  in  subsequent  chapters. 

Chapter  4:  Selecting  An  Evaluation  Approach.  Chapter  4  begins 
with  a  discussion  of  a  number  of  factors  which  the  ATD  OT&E  test  team 
must  consider  when  selecting  either  an  analytic  or  demonstration 
approach  appropriate  to  a  particular  ATD  OT&E  requirement  and/or  eval¬ 
uation  environment.  This  list  of  factors  includes  the  site  where  the 
evaluation  is  to  be  conducted;  the  calendar  time  available  to  complete 
the  evaluation;  the  level  of  management  commitment  to  the  evaluation; 
the  design  and  intended  use  of  the  ATD  to  be  evaluated;  the  availabil¬ 
ity  of  adequate  criterion  measures;  the  accessibility  of  the  ATD  and 
the  operational  equipment  for  evaluation  purposes;  the  availability  of 
instructor  pilots,  subjects,  and  other  necessary  resources;  and, 
finally,  the  characteristics  of  the  available  subject  population. 

After  these  factors  are  discussed,  Chapter  4  continues  by  pro¬ 
viding  general  guidance  to  the  test  director  and  his  team  on  how  to 
select  an  evaluation  approach,  using  this  information.  It  then  dis¬ 
cusses  how  to  apply  this  general  guidance  to  the  selection  of  an 
approach  within  a  particular  testing  environment.  Finally,  the  impor¬ 
tance  of  identifying  and  managing  user  attitudes  towards  the  ATD  to  be 
evaluated  is  discussed  and  guidance  on  how  to  do  so  is  provided. 

Chapter  4  concludes  by  pointing  out  that  Chapters  1  through  4, 
inclusive,  have  been  oriented  primarily  toward  establishing  a  basic 
background  for  an  understanding  by  the  test  team  of  the  planning, 
management,  training  program,  device,  measurement,  attitude,  and  other 
factors  with  which  they  must  be  familiar  as  they  proceed  with  the  con¬ 
duct  of  an  ATD  OT&E.  It  is  recognized  at  this  point  in  the  Handbook 
that  the  test  team  also  needs  very  specific,  detailed  instructions 
concerning  a  number  of  the  evaluation  tools  and  techniques  they  will 
employ  in  performing  either  an  analytic  or  an  applications  type  ATD 
OT&E .  As  a  result,  the  remaining  chapters  of  Volume  II  are  devoted  to 
the  presentation  of  such  information. 

Chapter  5:  Rating  Scales  and  Questionnaires.  This  chapter 

acquaints  the  test  director  with  the  techniques  he  will  need  in  order 
to  construct,  pre-test,  and  use  rating  scales  and  questionnaires 
during  an  ATD  OT&E.  The  chapter  opens  with  a  brief  discussion  of  the 
advantages  and  limitations  of  the  rating  scale  concept.  That  dis¬ 
cussion  is  followed  by  the  technical  body  of  the  chapter  which  is 
organized  into  three  major  sections. 
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Section  A  develops  the  rating  scale  concept  and  procedures  for 
rating  scale  construction  to  a  level  such  that  the  test  director  will 
have  the  background  necessary  to  effective  application  of  rating  scale 
methodology.  He  is  introduced  to  the  principles  of  rating  scale  con¬ 
struction  and  is  given  guidance  on  how  to  construct  many  types  of 
rating  forms. 

Section  B  of  this  chapter  describes  specific  rating  scale  proce¬ 
dures  he  may  use  in  conducting  (by  use  of  the  rating  method)  two 
important  elements  of  ATO  OT&E  evaluation:  fidelity  assessments  and 
training  capability  evaluations  of  the  crew  station.  It  should  be 
noted  here  that  the  rating  method  is  also  used  in  part  to  evaluate  the 
instructor/operator  station  (IOS).  However,  in  view  of  its  unique 
evaluation  requirements,  evaluation  of  the  IOS  has  been  addressed 
separately  from  the  scope  of  this  rating  scale  method  chapter.  The 
procedures  for  IOS  evaluation  are  provided  in  the  last  chapter  of 
Vol ume  1 1 ,  Chapter  7 . 

Section  C  of  this  chapter  points  out  that  questionnaires  are 
extensively  used  during  0T£E  because  they  are  relatively  inexpensive 
to  develop,  simple  to  administer,  and  allow  large  amounts  of  data  to 
be  gathered  quickly.  This  section  also  makes  the  point,  however,  that 
poorly  constructed  questionnaires  can  be  troublesome,  in  that  they  are 
likely  to  result  in  distorted  data  and  possibly  lead  one  to  unwar¬ 
ranted  conclusions.  In  order  to  minimize  the  likelihood  of  a  test 
director  using  improperly  prepared  questionnaires,  this  chapter  pro¬ 
vides  specific  guidance  on  the  construction  of  questionnaire  state¬ 
ments  and  procedures  for  their  formatting  and  organization. 

Chapter  6:  Transfer  of  Training.  This  chapter  constitutes  one 
of  the  most  important  elements  of  information  presented  in  Volume  II: 
It  provides  a  quantitative  method  for  ATD  OT&E  evaluation  which 
depends  not  on  expert  opinions  or  IP  judgments,  but  which  involves 
actual  ATD  training  activity. 

The  chapter's  technical  content  is  presented  in  two  sections. 
The  first  section  provides  guidance  concerning  the  general  conduct  of 
a  TOT  study  from  planning,  through  execution,  to  reporting.  In  anti¬ 
cipation  that  a  greater  degree  of  specificity  would  prove  of  use,  the 
second  section  of  Chapter  6  presents  four  types  of  TOT  designs  that  a 
test  team  might  find  applicable  under  many  ATD  OT&E  situations.  The 
breadth  and  depth  of  coverage  in  these  two  sections  is  such  that  the 
test  team  should  be  able  to  implement  an  effective  TOT. 

Chapter  7:  IOS  Evaluation.  Chapter  7  is  the  concluding  chapter 
of  Volume  II.  As  noted  earlier,  it  reflects  an  application  of  rating 
methodology  to  the  problem  of  ATD  IOS  evaluation  during  IOT&E/FOT&E. 
The  importance  of  the  IOS  to  effective  ATD  utilization  is  recognized 
early  in  the  chapter.  The  chapter  identifies  the  types  of  instructor/ 
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operator  tasks  likely  to  be  performed  at  an  IOS  and  relates  those 
tasks  to  the  specific  instructional  features  with  which  an  IOS  may  be 
equipped.  The  chapter  provides  specific  procedures  for,  and  aids  to, 
the  conduct  of  an  IOS  evaluation. 

Vol ume  III:  Operational  Suitability  Evaluation 

This  volume  is  concerned  with  assessing  the  impact  of  operational 
suitability  factors  on  ATD  usefulness.  Operational  suitability  fac¬ 
tors  pertain  to  how  well  the  device  meets  accepted  equipment  service¬ 
ability  reauirements  within  its  intended  operating  and  maintenance 
environment.  There  are  two  principal  components  of  ATD  suitability 
that  must  be  examined  during  ATD  OT&E,  i.e.,  hardware  suitability  and 
software  suitability.  Assessment  of  ATD  operational  suitability 
during  OT&E  is  an  important  area  of  concern,  and  one  requiring  great 
care  in  its  planning  and  execution.  This  volume  is  intended  princi¬ 
pally  for  the  purpose  of  helping  the  ATD  test  director  in  identifying 
the  appropriate  suitability  assessment  technical  personnel  and  of 
understanding  better  the  activities  they  perform.  Accordingly,  this 
volume  is  not  intended  to  be  a  substitute  for  technical  how-to 
manuals,  but  rather  an  information  source  and  test  planner/director 
guidebook  for  ATD  suitability  assessments  during  OT&E.  The  following 
are  brief  summaries  of  Volume  III  chapters: 

Chapter  1:  Purpose  and  Organization  of  Volume.  This  chapter 
provides  a  brief  introduction  to  the  concept  of  operational  suitabil¬ 
ity,  definitions  of  suitability  factors,  a  statement  of  the  intended 
informational  uses  of  the  volume,  and  a  roadmap  to  the  content  of  the 
vol ume. 

Chapter  2:  Hardware  Suitability.  This  chapter  presents  the 
major  hardware  suitability  evaluation  concerns  of  reliability,  main¬ 
tainability,  availability,  logistics  supportabil  ity ,  and  operating  and 
support  costs.  For  each  major  suitability  element,  discussions  are 
provided  that  define  that  element  in  the  context  of  ATD  OT&E.  In 
addition,  under  the  heading  of  "Test  Director  Concerns,"  generic  per¬ 
sonnel  requirements  are  identified  and  practical  guidelines  are  pro¬ 
vided  that  may  serve  to  inform  the  test  director  of  possible  lead  time 
requirements,  and  phase  of  test  specific  concerns  to  aid  in  the  suc¬ 
cessful  accomplishment  of  the  various  operational  suitability 
assessments. 

Chapter  3:  Software  Suitability.  This  chapter  provides  an 
introduction  to  software  suitability  assessments  for  ATD  OT&E.  Soft¬ 
ware  suitability  is  defined  in  terms  of  maintainability  and  usability. 
It  is  noted  that  the  technology  of  software  evaluations  is  a  currently 
evolving  area  requiring  access  to  appropriate  evaluation  personnel. 
The  content  of  the  chapter  corresponds  to  the  evaluation  elements  of 
maintainabil  i ty  and  usability  and  defines  the  subelements  under  each 
along  with  a  description  of  evaluation  procedures  carried  out  by 
qualified  software  evaluation  personnel. 


CHAPTER  2 


UNDERSTANDING  ATD  ACQUISITION  AND  TEST 


INTRODUCTION 

Organization  and  management  of  an  effective  test  program  requires 
that  the  test  director  and  his  team  have  a  basic  understanding  of  the 
role  of  test  and  evaluation  (T&E)  in  ATD  acquisition  and  employment. 
This  chapter  provides  the  background  information  necessary  for  that 
understanding.  There  are  two  major  topics  covered:  (1)  ATD  acquisi¬ 
tion  procedures;  and  (2)  ATD  OT&E  concepts  and  responsibilities.  The 
discussion  of  the  ATD  acquisition  process  is  intended  to  provide  the 
OT&E  test  team  a  summary  of  the  major  phases  in  the  procurement  of  an 
ATD.  It  is  provided  mainly  to  familiarize  the  test  team  with  the  ATD 
acquisition  process  and  its  associated  documentation.  As  such,  it  is 
not  intended  as  a  detailed  guide  for  procurement  or  test  and  evalua¬ 
tion  staff  activities.  Detailed  procedures  for  such  activities  may  be 
found  in  the  internal  documentation  of  the  organizations  and  staff 
agencies  responsible  for  these  functions. 

The  second  topical  discussion  begins  with  a  description  of  those 
early  OT&E  planning  activities  that  go  on  simultaneously  with  ATD 
acquisition-  It  concludes  with  an  identification  of  the  various 
types  of  ATD  OT&E,  their  respective  goals,  and  organizational 
respon sibil ities. 


THE  ATD  ACQUISITION  PROCESS 
Program  Initiation 


Acquisition  of  aircrew  training  devices  by  the  Air  Force  is  han¬ 
dled  by  the  Deputy  for  Simulators  (Simulator  System  Program  Office — 
SimSPO)  of  the  Aeronautical  Systems  Division,  Air  Force  Systems 
Command  (ASD/AFSC).  The  SimSPO  follows  established  procedures  for  ATD 
procurement  planning,  scheduling,  and  monitoring  from  ATD  project 
inception  through  contract  award,  qua! i fication/acceptance  testing, 
and  final  transfer  of  the  ATD  to  the  using  command  and  the  Air  Force 
Logistics  Command  (AFLC). 

Acquisition  of  a  new  ATD  is  initiated  by  a  Statement  of  Opera¬ 
tional  Need  (SON),  or  a  Mission  Essential  Needs  Statement  (MENS). 
Either  document  is  originated  by  one  of  the  Air  Force  Major  Commands 
( MAJCOM) .  An  ATD  SON  or  MENS  is,  by  necessity,  a  very  general  state¬ 
ment  of  requirements.  In  fact,  the  ATD  portion  of  the  pertinent  docu¬ 
ment  for  a  new  weapon  system  (e.g.,  F-16)  may  be  simply  a  statement 
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that  associated  training  devices  would  be  required.  During  the  SON/ 
MENS  process,  the  using  command  and  AFSC  work  together  to  define 
realistic  ATD  solutions  to  the  expressed  need  prior  to  sending  a  final 
evaluation  (including  a  recommended  solution)  of  the  SON/MENS  to  USAF 
for  approval  and  further  direction. 

HQ  USAF  provides  program  management  direction  and  guidance  at  the 
outset  of  the  procurement  cycle  by  issuing  a  Program  Management 
Directive  (PMD).  The  PMD  reflects  approval  of  the  program  by  HQ  USAF 
and  specifies  the  actions  required  to  translate  the  SON/MENS  into  a 
proposal  for  a  new  program.  At  this  point,  the  program  is  assigned  to 
the  SimSPO,  and  a  program  manager  is  appointed.  The  PMD  is  also  for¬ 
warded  to  the  using  command,  the  Air  Force  Test  and  Evaluation  Center 
(AFTEC),  the  Air  Force  Logistics  Command  (AFLC),  and  other  partici¬ 
pating  commands  or  agencies  so  that  necessary  advance  planning  may  be 
initiated. 

Proposal  Phase 


Procurement  activity  begins  with  the  Request  for  Proposal  (RFP). 
Preparation  of  the  Specification  and  Statement  of  Work  (SOW)  to  be 
contained  in  the  RFP  is  the  process  wherein  the  SimSPO  and  using  com¬ 
mand  come  to  a  mutual  understanding  as  to  the  desired  ATD  configu¬ 
ration.  A  draft  of  the  RFP  may  be  issued  to  industry  for  comment  as 
part  of  this  process.  A  synopsis  of  the  program  is  prepared  and 
published  in  the  Commerce  Business  Daily  (CBD)  so  that  all  ATD  manu¬ 
facturers  who  possess  appropriate  capabilities  may  request  copies  of 
the  final  RFP.  Copies  of  the  final  RFP  are  then  distributed  to  indus¬ 
try  in  accord  with  the  government's  view  of  the  qualifications  of  the 
•various  interested  vendors. 

Upon  receiving  the  RFP,  the  responding  ATD  manufacturers  prepare 
detailed  technical  proposals  describing  their  individual  design  con¬ 
cepts  for  the  subject  ATD,  and  their  program  management  and  scheduling 
plans.  In  addition  to  the  technical  proposal ,  each  manufacturer  pre¬ 
pares  a  cost  proposal  which  contains  that  offeror's  bid  for  the  design 
and  production  of  the  device.  At  a  date  specified  in  the  RFP,  vendor 
proposals  are  submitted  to  the  Air  Force  for  evaluation. 

A  systematic  evaluation  of  offerors'  proposals  is  then  carried 
out  to  select  from  among  those  submitting  proposals  those  firms  that 
may  be  considered  for  award  of  the  contract.  Evaluations  consider 
both  technical  responsiveness  to  the  RFP  and  price  factors.  One  or 
more  offerors  judged  to  be  within  the  competitive  range  may  then  be 
invited  to  negotiate  further  or  clarify  proposals  on  technical  and/or 
price  factors,  and  to  submit  a  best  and  final  offer  (BAFO)  to  the 
government. 

After  receipt  of  the  best  and  final  offers,  a  Source  Selection 
Committee  conducts  a  final  evaluation  and  awards  the  ATD  contract  to 
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one  of  the  offerors.  In  some  cases,  more  than  one  contract  may  be 
awarded,  and  a  "fly  off"  between  contractors  may  be  conducted.  In 
such  instances,  a  final  source  selection  would  be  based  upon  test  and 
evaluation  results  of  the  prototype  devices. 

Production  Phase 


The  production  phase  consists  of  three  major  acquisition  efforts: 
Preliminary  Design,  Detailed  Design,  and  Fabrication  and  Acceptance 
Testing.  Contract  specifications,  the  Statement  of  Work  (SOW),  and 
the  Program  Schedule  are  key  documents  for  understanding  precisely 
what  is  to  be  built  and  when  OT&E-rel evant  acquisition  milestones 
should  occur. 

Preliminary  design  phase.  During  this  phase  the  ATD  contractor 
conducts  a  detailed  review  of  the  specification  and  develops  a  prel im- 
inary  design  approach.  The  resulting  preliminary  design  is  then  pre¬ 
sented  to  the  Air  Force  during  a  Preliminary  Design  Review  (PDR).  As 
part  of  the  PDR,  a  mock-up  review  is  usually  held.  During  mock-up 
review  the  Air  Force  customer  may  examine  a  nonworking  model  of  the 
device  (this  may  include  both  trainee  station  and  instructor/operator 
station).!  The  PDR  provides  an  opportunity  for  Air  Force  inputs 
regarding  device  design  and  conformance  to  contractual  requirements, 
including  requirements  related  to  the  trainee  station,  instructor  sta¬ 
tion,  computational  system,  math  model,  motion  system,  visual  system, 
safety  system,  reliability,  maintainability,  etc.  Air  Force  attendees 
at  PDR  usually  include  SimSPO,  AFTEC  (OT&E  team),  and  using  command 
representatives.  Representatives  from  other  participating  commands 
and  agencies  may  also  be  present. 

Detailed  design  phase.  During  this  phase  specific  details  of  the 
ATD  are  addressed  to  arrive  at  actual  hardware  production  drawings, 
and  computer  processing  functions  are  flow-charted.  This  design 
activity  incorporates  the  Air  Force  input  received  during  PDR  and 
mock-up  review.  Periodic  design  reviews  are  held  to  ensure  that  all 
subsystems  are  designed  to  interface  correctly.  Software  module 
development  and  testing  is  begun,  and  hardware/ software  interface 
requirements  are  defined.  At  the  end  of  the  detailed  design  phase  a 
Critical  Design  Review  (CDR)  is  held  to  allow  the  Air  Force  to  review 
the  detailed  design  of  the  ATD.  At  this  point  design  discrepancies 
identified  during  PDR  are  reviewed  to  ensure  that  they  have  been 
corrected  before  hardware  manufacture  begins. 

Fabrication  phase.  This  phase  consists  of  a  number  of  activi¬ 
ties:  hardware  fabrication,  software  development,  and  systems 


Mock-up  may  occur  independently  of  PDR.  If  so,  OT&E  team  repre¬ 
sentation  is  appropriate. 


integration.  Hardware  fabrication  includes  manufacture  of  certain 
hardware  items  as  well  as  integration  of  purchased  hardware  (indi¬ 
cators,  display  systems,  etc.)  with  manufactured  hardware.  All  hard¬ 
ware  contained  in  major  ATO  subsystems,  i.e.,  trainee  station, 
instructor/operator  station,  visual  system,  motion  system,  computa¬ 
tional  system,  etc.,  is  completely  assembled  into  the  ATD.  Critical 
functional  software  modules  developed  include  trainee  station  simula¬ 
tion  drive  modules  (e.g.,  flight  characteristics,  tactical  environ¬ 
ment,  target  characteristics,  emergency  procedures)  and  instructor/ 
operator  station  modules  (e.g.,  problem  set  up,  performance  moni¬ 
toring,  instructional  features). 

System  integration  and  checkout  occurs  when  all  ATD  subsystems 
(including  hardware  and  software)  are  fully  integrated.  This  is  often 
referred  to  as  Hardware/ Software  Integration  (HS1).  Verification 
tests,  an  in-house  quality  control  activity,  are  then  conducted.  They 
involve  using  the  preliminary  QTPs  to  test  major  systems  and  sub¬ 
systems.  Any  deficiencies  that  may  be  identified  are  then 
corrected.! 

Development  of  Qualification  Test  Procedures  (QTP)  also  occurs 
during  this  phase.  A  preliminary  set  of  QTPs  is  prepared  and  sub¬ 
mitted  to  the  Air  Force,  usually  120  -  180  days  prior  to  the  start  of 
formal  testing.  A  repetitive  review,  correction,  and  revision  process 
is  used  in  the  development  of  the  QTP  that  will  be  followed  during  the 
test.  A  final  QTP  is  delivered,  under  the  contract,  prior  to  testing. 

Following  verification  testing  activities  and  submission  of  the 
final  QTP  (and  assuming  any  deficiencies  found  are  corrected),  the 
ATD  is  declared  "ready  for  Air  Force  test  and  acceptance." 

Acceptance  Test  and  Eva! uation  (AT&E)  Phase 

This  final  phase  of  activity  in  the  ATD  acquisition  process 
includes  qual ification/acceptance  tests  by  the  Air  Force  to  demon¬ 
strate  that  the  engineering  design  is  complete,  that  design  and 
production  risks  are  minimized,  and  that  the  requirements  and  speci¬ 
fications  of  the  procurement  contract  are  fulfilled.  Completion  of 
this  phase  is  formalized  by  DO  250  signoff. 

Economic  Analyses  During  ATD  Acqui si ti on 

Economic  analyses  of  military  systems  are  required  by  Department 
of  Defense  Instruction  (DoDI)  7041.3,  "Economic  Analysis  and  Program 


^Contractor  verification  testing  is  often  monitored  by  the 
Defense  Contracts  Administration  Service  (DCAS)  to  validate  the 
contractor's  readiness  for  test. 
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Evaluation  for  Resource  Management."*  DoDI  7041.3  states  as  policy 
that  project  officers  and  managers  should  be  prepared  to  demonstrate 
the  cost-effectiveness  of  budget  proposals  and  to  submit  detailed 
analyses  in  support  of  budget  estimates  provided  in  accord  with  DoD 
7110-1-M,  "Department  of  Defense  Budget  Guidance  Manual."  In  devel¬ 
oping  and  justifying  resource  requirements,  an  economic  analysis  is 
required  for  budget  proposals  which  involve  a  choice  or  trade-off  be¬ 
tween  two  or  more  options,  even  when  one  of  the  options  is  to  maintain 
the  status  quo  or  to  do  nothing. 

Such  analyses  are  particularly  important  during  ATD  acquisition 
as  evidenced  by  those  economic  analyses  which  were  conducted  for  the 
B-52,  C-130,  F-15,  and  F-16  ATD  programs. 

Definition  of  economic  analysis.  DoDI  7041.3  defines  an  economic 
analysis  as  "a  systematic  approach  to  the  problem  of  choosing  how  to 
employ  scarce  resources  and  an  investigation  of  the  full  implications 
of  achieving  a  given  objective  in  the  most  efficient  and  effective 
manner."  Determinations  of  efficiency  and  effectiveness  are  implicit 
in  assessment  of  the  cost  effectiveness  of  alternative  approaches. 

Such  assessments  during  ATD  acquisition  are  accomplished  by: 

1.  Identifying  systematically  the  benefits,  other  outputs,  and 
costs  associated  with  alternative  training  program  concepts. 

2.  Identifying  the  assumptions  on  which  ATD  procurement  and 
employment  decisions  are  based,  including  technical,  opera¬ 
tional  ,  schedule,  and  other  performance  considerations. 

3.  Evaluating  alternative  methods  of  supporting  an  ATD  acquisi¬ 
tion,  e.g.,  lease  or  buy;  contractor  vs.  Air  Force  main¬ 
tenance. 

4.  Using  benefits  and  costs  to  compare  the  relative  merits  of 
relevant  alternatives  as  an  aid  in  recommending  the  preferred 
alternative. 

Scheduling  of  economic  analyses.  DoDI  7041.3  specifies  that  eco¬ 
nomic  analyses  are  to  be  initiated  as  early  in  the  acquisition  process 
as  practical  and  updated  whenever  significant  developments  occur  which 
could  invalidate  or  signi ficantly  alter  the  cost-benefit  relationships 
upon  which  previous  decisions  were  made.  DoDI  7041.3  also  calls  for 


*DoDI  7041.3  has  been  implemented  in  the  Air  Force  through  Air 
Force  Regulation  (AFR)  178-1,  which  essentially  is  a  paraphrase  of 
DoDI  7041.3. 
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periodic  program  evaluations  to  assess  the  cost  effectiveness  of 
ongoing  activities. 

DoDI  7041.3  provides  for  three  exceptions  to  the  requirement  for 
economic  analysis.  An  economic  analysis  is  not  required: 

1.  When  it  can  be  shown  that  cost  of  the  minimum  level 
of  effort  required  to  do  the  analysis  would  exceed 
the  worth  of  the  benefits  to  be  gained  from  such  an 
analysis. 

2.  In  cases  where  OoD  instructions  and  issuances  pre¬ 
scribe  equipment  or  age  replacement  criteria,  labor 
and  equipment  trade-off  standards,  or  requirements 
computations  which,  in  turn,  have  been  based  on  an 
economic  analysis  as  called  for  in  the  DoD  instruc¬ 
tion. 

3.  When  proposed  actions  are  specifically  directed  by 
legislation  or  prior  irrevocable  management  deci¬ 
sions  which  preclude  any  choice  or  trade-off  among 
alternative  ways  to  accomplish  a  program/project. 

ATP  economic  analysis  models.  The  economic  analysis  models  cur¬ 
rently  available  to  the  Air  Force  for  use  during  ATD  acquisition  are 
not  true  cost-effectiveness  models,  in  that  they  do  not  allow  a  real 
evaluation  of  the  effects  of  variations  in  training  effectiveness. 
The  models  generally  in  use  treat  effectiveness  as  "fixed."  The 
assumption  is  made  in  each  of  these  models  that  the  training  alter¬ 
natives  of  interest  are  equally  effective,  and  that  simple  cost  com¬ 
parisons  will  suffice  for  cost-effectiveness  analysis  purposes. 

There  are  two  such  cost  models  actively  utilized  by  the  SimSPO 
during  the  ATD  acquisition  cycle:  the  RCA  Cost  Model  and  the  Logistic 
Support  Cost  Impact  Model . 

RCA  Cost  Model.  The  RCA  Cost  Model^  is  utilized  by  the  Air 
Force  Simulator  System  Program  Office  (SimSPO)  to  estimate  simulator 
hardware  costs.  For  each  simulator  subsystem  the  contractor  provides 
parametric  hardware  information  such  as  size,  weight,  mechanical/ 
electric  interfaces,  electronic  packing  densities,  numbc  of  electro¬ 
nic  parts,  material  used  in  structure,  method  of  cooling,  weight  and 
volume  of  discrete  electronic  modules,  etc.  Based  on  this  infor¬ 
mation,  the  model  calculates  a  hardware  system  cost  estimate.  The 


1 


Details  are  proprietary  to  RCA. 
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model  is  used  by  the  SimSPO  primarily  to  provide  an  "independent" 
estimate  (relative  to  that  of  the  contractor)  of  simulator  hardware 
costs  to  support  management  decisions  which  must  be  made  during  the 
early  phases  in  procurement  of  an  ATD. 

Logistic  Support  Cost  Impact  Model.  The  Logistic  Support 
Cost  Impact  Model  (LSC)  is  used  by  the  SimSPO  to  estimate  the  oper¬ 
ating  and  support  ( O&S )  costs  that  may  be  incurred  by  adopting  a  spe¬ 
cific  system/product  design  configuration.  This  model  is  used  in  two 
ways:  (1)  as  a  decision  aid  when  discriminating  among  design  alter¬ 

natives  during  the  advanced  development  and  detailed  design  phases  for 
an  ATD;  and  (2)  to  estimate  the  differential  logistics  support  costs 
between  the  proposed  design  configurations  of  two  or  more  potential 
ATD  suppliers  during  source  selection. 


ATD  T&E  CONCEPTS  AND  RESPONSIBILITIES 

There  are  two  primary  categories  of  ATD  T&E:  Development  Test 
and  Evaluation  (DT&E)  and  Operational  Test  and  Evaluation  (OT&E). 
DT&E  is  conducted  principally  to  determine  that  the  ATD  meets  the 
engineering  design  and  development  specifications.  OT&E  is  conducted 
to  assess  the  training  utility  of  the  ATD,  and  to  develop  estimates  of 
its  operational  suitability.  Funding  for  T&E  comes  from  either  RDT&E 
funds,  or  from  Operations  and  Maintenance  ( O&M)  funds. 

Development  Test  and  Evaluation  (DT&E) 


DT&E  is  that  test  and  evaluation  conducted  to  demonstrate  that 
ATD  engineering  design  and  development  are  complete,  that  design  risks 
have  been  minimized,  and  that  the  system  will  meet  engineering  and 
operational  specifications.  DT&E  involves  essentially  a  detailed 
engineering  analysis  of  the  ATD's  performance,  beginning  with  indivi¬ 
dual  subsystems  and  progressing  through  a  complete  system.  System 
design  is  tested  and  evaluated  against  engineering  and  performance 
criteria  provided  by  the  Air  Force. 

The  SimSPO  is  responsible  for  management  of  DT&E.  AFTEC  and  the 
MAJCOMs  participate  in  DT&E  as  specified  in  the  program  directives  and 
in  the  coordinated  planning  documents.  DT&E,  a  natural  part  of  the 
contractor  development  process,  is  initiated  as  early  in  the  develop¬ 
ment  cycle  as  possible  and  includes  testing  of  component! s) ,  sub¬ 
system!  s),  and  prototype  or  preproduction  model (s)  of  the  entire 
system. 

DT&E  for  systems  where  there  is  no  development  funding  is  known 
as  Qual ification  Testing  (QT&E).  As  with  regular  DT&E,  qualification 
tests  are  conducted  to  demonstrate  that  engineering  design  is  com¬ 
plete,  that  design  and  production  risks  are  minimized,  and  that  the 
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items  fulfill  the  requirements  and  specifications  of  the  procuring 
contract  or  agreement.  QT&E  is  managed  and  conducted  in  a  manner 
similar  to  DT&E. 

Operational  Test  and  Evaluation  (OT&E) 

Operational  Test  and  Evaluation  ( OT&E )  is  divided  into  two 
subsets:  IOT&E  and  FOT&E.  IOT&E  is  conducted  to  provide  an  initial 

estimate  of  the  operational  training  effectiveness  and  operational 
suitability  of  the  device,  to  identify  operational  deficiencies,  and 
tc  suggest  appropriate  changes  to  overcome  those  deficiencies.  IOT&E 
is  accomplished  by  test  teams  consisting  of  Air  Force  operational  and 
support  personnel  representative  of  those  who  are  expected  to  use  and 
maintain  the  ATO  when  it  is  deployed.  Either  AFTEC  or  the  using 
MAJCOM  (AFTEC  monitored)  may  be  responsible  for  IOT&E. 

IOT&E  may  be  combined  with  DT&E.  When  combined,  testing  may 
include  shared  DT&E/IOT&E  test  events  and  separate  DT&E  and  IOT&E  test 
events.  It  should  be  noted  that  combined  testing  does  not  mean  that 
one  type  of  testing  predominates  or  that  all  IOT&E  objectives  can  or 
will  be  satisfied  by  shared  test  events.  It  should  be  noted  that 
Qualification  Operational  Test  and  Evaluation  (QOT&E)  is  conducted  in 
lieu  of  IOT&E  for  ATDs  where  there  has  not  been  any  RDT&E  funding. 
QOT&E  is  functionally  equivalent  tc  IOT&E  with  regard  to  test  objec¬ 
tives  and  procedures.  The  funding  source  is  the  only  differentiating 
factor. 

Follow-on  Operational  Test  and  Evaluation  (FOT&E) 

FOT&E,  that  test  and  evaluation  conducted  after  IOT&E/QOT&E,  is 
normally  conducted  in  two  phases.  Phase  I  FOT&E  is  conducted  to 
refine  and  expand  assessments  made  during  IOT&E.  Further,  it  confirms 
that  corrections  for  previously  noted  operational  deficiencies  work 
and  it  ensures  that  the  implemented  device  performs  as  predicted  based 
upon  experience  gained  during  earlier  T&E.  AFTEC  is  responsible  for 
managing  Phase  I  FOT&E  on  major  and  designated  nonmajor  programs. 
Otherwise,  Phase  I  OT&E  is  the  responsibility  of  the  using  MAJCOM. 
Phase  II  FOT&E  is  normally  conducted  by  the  using  command,  its  pur¬ 
pose  is  to  develop  and  refine  further  the  utilization  of  the  ATD 
within  its  specific  training  context.  FOT&E  programs  are  supported  by 
the  using  MAJCOM' s  O&M  budget. 
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CHAPTER  3 


PLANNING  AND  MANAGEMENT  OF  ATD  OT&E 


INTRODUCTION 

The  test  and  evaluation  of  ATDs  in  the  Air  Force  requires 
involvement  of  a  number  of  organizations  and  agencies.  In  some 
instances  that  involvement  is  one  of  direct  management  responsibility 
and  participation,  while  in  others  a  support  role  is  reouired.  The 
general  nature  of  T&E  involvement  and  responsibility  among  Air  Force 
organizations  is  as  follows: 

HQ  USAF 

HQ  USAF:  (1)  provides  T&E  direction  via  PMDs  and  test  direc¬ 
tives;  (2)  establishes,  and  publishes  in  appropriate  documents,  the 
initial  critical  questions  and  areas  of  risk  which  are  then  sub¬ 
sequently  refined  by  the  implementing  command,  the  participating  com¬ 
mands,  and  AFTEC;  (3)  designates  the  agencies  or  commands  to  be 
responsible  for  specific  test  and  evaluation  programs,  including  the 
extent  of  AFTEC  and  MAJCOM  participation  in  OT&E;  (4)  reviews  test 
requirements  and  approves  the  allocation  of  HQ  USAF  controlled 
resources;  (5)  provides  instructions  for  the  disposition  and  support 
of  test  articles  in  the  PMD  before  the  production  decision;  (6) 
resolves  any  inter-command  differences  which  may  exist  concerning  T&E; 
and  (7)  finalizes  inter-service  agreements,  as  required. 

AFTEC 


The  Air  Force  Test  and  Evaluation  Center:  (1)  plans,  directs, 
conducts,  controls  and  independently  evaluates  (and  reports  to  the 
Chief  of  Staff  of  the  Air  Force}  major  OT&E  programs  and  USAF  desig¬ 
nated  nonmajor  programs;  (2)  accomplishes  detailed  planning  and 
budgeting  for  OT&E’  (3)  participates  with  AFSC  in  preparation  of  the 
TEMP;  (4)  provides  information  on  operational  deficiencies  to  the 
system  program  manager  and  to  the  affected  MAJCOM;  and  (5)  monitors 
OT&Es  conducted  by  using  commands. 

AFSC 


As  the  implementing  command,  AFSC:  (1)  plans  programs,  and 

defines  the  scope  and  concept  of  DT&E;  (2)  exercises  final  responsi¬ 
bility  for  the  conduct  of  DT&E;  (3)  identifies  the  critical  questions 
and  areas  of  risk  to  be  addressed  as  test  objectives  during  T&E:  (4) 
develops  and  plans  the  T&E  program  to  meet  the  program  decision  mile¬ 
stones;  (5)  prepares  the  Test  and  Evaluation  Master  Plan  (TEMP), 


d( ‘■ailed  combined  test  plans,  and  other  program  documents;  (6)  plans 
and  budgets  for  reouired  update  or  modification  of  test  articles;  (7) 
collects,  analyzes,  and  evaluates  test  data  and  prepares  and  distri¬ 
butes  reports  on  DT&E;  and  (8)  provides  pre-operational  support. 

The  AFSC  SimSPO.  The  AFSC  program  manager  is  the  SimSPO.  The 
SimSPO  has  responsibility  for  the  ATO  acauisition  program.  The  SimSPO 
test  management  responsibilities  include  managing  DT&E.  incorporating 
the  OT&E  requirements  into  the  test  program,  and  providing  support  for 
ATD  OT&E.  AFTEC,  or  a  designated  MAJCOM,  not  the  SimSPO,  has  the 
responsibility  for  managing  ATD  OT&E. 

The  AFSC  AFHRL.  The  Air  Force  Human  Resources  Laboratory  (AFHRl) 
of  AFSC  is  a  behavioral  science  research  organization  that  can  provide 
technical  assistance  in  development  of  the  test  design  and  evaluation 
procedures  for  assessment  of  ATD  training  effectiveness  durinq  OT&E. 
That  assistance  may  include  preparation  of  detailed  operational  effec¬ 
tiveness  auestionnai res  and  rating  scales,  monitoring  of  test  progress 
to  assure  validity  of  training  effectiveness  data,  design  of  transfer 
of  training  studies,  and  preparation  of  training  effectiveness  por¬ 
tions  of  the  test  report. 

There  are  no  fixed  procedures  whereby  the  participation  of  AFHRL 
can  be  arranged.  As  a  result,  AFHRL  support  requirements  must  be 
identified  as  early  as  practicable  and  formally  negotiated  with  the 
AFHRL  commander. 

MAJCOMs 


The  MAJCOM  designated  to  use  the  system:  (1)  provides  operQ - 
tional  employment  and  maintenance  concepts  for  T&E,  (c)  identifies 
critical  Questions,  areas  of  risk,  test  objectives,  c  o  dat>;  '?quire- 
ments  in  the  operations  and  logistics  areas  for  ;  (3)  plant,  bud¬ 
gets,  and  provides  (from  within  the  command)  the  resources  (personnel, 
equipment,  flying  hours,  and  fuel)  necessary  to  accomplish  OT&E;  (4) 
participates  early  in  the  planning  and  programming  of  T&E;  (5)  manages 
IOT&E  when  designated;  and  (6)  manages  F0T3E. 

AFLC 


The  Air  Force  Logistics  Command:  (1)  provides  logistic  support 
and  planning  for  test  programs;  (?)  identifies  critical  auestions, 
areas  of  risk,  test  objectives  and  data  requirements  in  logistics 
areas  for  new  systems  under  T&E  and  (3)  participates  on  test  teams 
for  planning,  conducting,  and  determining  logistics  supportabil ity  for 
DT&E,  and  verifying  or  assessing  these  factors  for  OT&E. 


The  Air  Training  Command  acts  as  a  supporting  command  for  the 
responsible  OT&E  command  or  agency  by  evaluating  whether  Air  Force 
personnel  with  system  training  can  operate,  maintain,  and  support  the 
system  as  required  by  the  operational  and  maintenance  concepts.  ATC : 
(1)  provides  for  the  training  of  Air  Force  personnel  to  support  the 
test  program;  (2)  provides  OT&E  objectives  relative  to  the  evaluation 
of  training  for  maintenance  personnel;  (3)  participates  in  OTftE  test 
planning  conferences;  (4)  coordinates  on  OT&E  test  plans;  (5)  provides 
maintenance  training  inputs  to  the  operational  suitability  annex  of 
the  test  plan;  (6)  provides  evaluators  to  test  teams  to  accomplish 
maintenance  training-rel ated  test  and  evaluation  objectives;  (7)  par¬ 
ticipates  in  preparation  of  OT&E  test  reports;  and  (8)  provides 
training  costs  for  cost  of  ownership  studies. 

PHASES  OF  OT&E  ACTIVITY 

The  process  of  ATD  OT&E  planning  and  management,  may  be  conve¬ 
niently  partitioned  into  three  phases  of  activity — Planning,  Execu¬ 
tion,  and  Reporting.  Each  phase  corresponds  to  a  logical  separation 
of  functional  activity.  The  Planning  Phase  includes  preparation  of 
the  test  plan  and  precoordination  among  test  elements  and  resources  to 
carry  out  that  plan.  The  Execution  Phase  includes  the  active  collec¬ 
tion  of  data.  Finally,  the  Reporting  Phase  includes  data  reduction 
and  interpretation  and  preparation  of  requisite  reports  and  briefings 
as  to  the  results  of  the  OT&E. 

The  presentation  that  follows  corresponds  to  the  three  logical 
phases  identified  above  and  has  been  formatted  in  accord  with  the 
diagram  symbols  shown  below: 


Indicates  the  beginning  of  activity 


Indicates  a  preparation  activity 
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Indicates  a  process 


Indicates  data 


Indicates  a  decision  point 


Indicates  a  briefing  or  review 


Indicates  a  document/ report 


Indicates  the  termination  of  activity 


( BEGIN 
V  PLANNING  PHASE 


) 


PLANNING  PHASE 


ACQUIRE  AND 
REVIEW  OT&E 
GUIDANCE 
DOCUMENTS 


thi s  regard 
Eval uation . 
i  information 
throughout 
ance  document 


The  test  director  must  first  acquire  and  review 
those  OT&E  guidance  documents  and  related 
materials  which  will  be  needed  in  planning  and 
managing  the  test.  Of  particular  importance  in 
is  ATM  55-43,  "Management  of  Operational  Test  and 
"  In  two  volumes,  AFM  55-43  contains  a  wealth  of 
and  guidelines  for  OT&E  planning  and  management 
the  Air  Force.  It  is,  in  fact,  the  basic  guid- 
to  be  followed. 


As  a  supplement  to  AFM  55-43,  major  commands  may  set  specific 
command  policies  and  procedures  for  OT&E  planning  and  manage¬ 
ment.  If  involved  in  a  MAJCOM-managed  test,  the  appropriate 
MAJCOM  regulations  and  manuals  also  should  be  reviewed. 
These  include: 

•  TAC:  USAFTAWCR  55-8,  "Operational  Test  and 

Evaluation  Management  Procedures" 

•  MAC:  MACR  55 -80,  "Test  and  Eval uation" 

•  SAC:  SACR  55-57,  "Operational  Test  and 

Eval uation" 


In  addition  to  these  relevant  Air  Force  guidance  documents, 
the  test  director  should  thoroughly  familiarize  himself  with 
the  contents  of  this  ATD  OT&E  Handbook.  This  is  important 
because  neither  AFM  55-43  or  the  above  referenced  MAJCOM 
materials  are  ATD  OT&E  specific,  but,  instead,  deal  with 
general  OT&E  of  any  system.  Many  of  the  special  considera¬ 
tions  that  must  be  taken  into  account  when  dealing  with  ATD 
OT&E  are  not  explicitly  covered  in  these  Air  Force  documents, 
e.g.,  methods  for  the  assessment  of  training  utility  of  ATDs. 
The  test  director  should  also  assure  that  his  test  team  per¬ 
sonnel  become  familiar  with  these  guidance  materials. 
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Review  of,  and  familiarization  with,  existing 
program  documentation  can  provide  the  test 
director  a  basic  under  standing  of  the  history  of 
the  program,  its  anticipated  schedule,  and  the 
nature  of  involvement  among  organi zations  in  the  test  and 
evaluation  of  the  ATD.  At  a  minimum,  the  following  documen¬ 
tation  should  exist  for  an  I OTS E/QOT & E  program  and  should  be 
acquired  and  referenced. 

•  Program  Management  Directive  (PMD) 

Upon  receipt  of  the  PMD,  a  Test  Planning  Working  Group 
(TPWG)  is  established  by  the  system  program  manager.  The 
TPWG  will  normally  include  representatives  from  the  SimSPO, 
AFSC  test  agencies,  AFTEC,  using  command,  AFLC,  ATC,  and 
other  agencies  to  be  involved  in  the  test  program.  The 
purpose  of  the  TPWG  is  to  provide  a  forum  for  test  related 
subjects,  to  assist  in  establishing  test  objectives  and 
evaluation  baselines,  to  define  organizational  responsi¬ 
bilities  and  relationships,  to  estimate  test  costs  and  sche¬ 
dules,  and  to  identify  required  test  resources.  The  TPWG 
assists  the  ATD  program  manager  in  preparing  the  Test  and 
Evaluation  Master  Plan  (TEMP),  the  Program  Management  Plan 
( PMP) ,  and  other  test  related  documents. 
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•  Test  and  Evaluation  Master  Plan  (TEMP) 

The  Test  and  Evaluation  Master  Plan  (TEMP)  is  an  overall 
test  and  evaluation  plan  designed  to  identify  and  integrate 
the  effort  and  schedules  of  all  T&E  to  be  accomplished  in  the 
ATD  acquisition  program.  The  TEMP  reflects  an  agreed-upon 
T&E  approach,  an  overall  summary  for  upper  management  of 
planned  T&E,  and  a  framework  for  the  more  detailed  test 
planning  to  follow.  The  TEMP  should  ensure  that  all  neces¬ 
sary  test  related  activities  are  planned  before  key  program 
decision  points.  In  contrast  to  the  management  orientation 
of  Section  5  of  the  PMP  (discussed  above),  the  TEMP  empha¬ 
sizes  test  objectives,  test  methodology,  test  measurement, 
and  test  resources  to  be  required.  It  should  specify 
expected  methods  of  testing  to  verify  these  objectives, 
measures  of  system  performance,  data  requirements,  resource 
requi rements ,  and  data  reduction  and  analysis  requirements. 
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•  Program  Management  Plan  (PMP) 

The  Program  Management  Plan  (PMP)  is  the  principal  manage¬ 
ment  baseline  document  for  the  program.  The  document  is 
written  and  issued  by  the  ATD  program  manager  and  it  shows 
the  integrated,  time-phased  tasks  and  resources  required  to 
accomplish  the  requirements  of  the  PMD.  Preparation  of  the 
PMP  requires  considerable  cooperation  and  coordination  of 
other  major  commands  and  organizations,  such  as  AFLC,  ATC, 
using  command,  and  AFTEC.  Section  5  of  the  PMP  emphasizes 
the  test  management  relationships,  resources,  basic  T&E 
objectives,  test  milestones,  and  schedules. 

•  Test  Program  Outline  (TPO) 

AFTEC ' s  fundamental  document  for  OT&E  planning  and 
budgeting  is  the  Test  Program  Outline  (TPO).  It  is  normally 
written  after  management  responsibilities  for  the  OT&E 
program  are  identified.  The  TPO  is  used  for  inputs  to  the 
budget,  PMP,  TEMP,  and  system  Test  Plan.  The  document  gener¬ 
ally  includes  the  basic  OT&E  issues,  objectives,  critical 
questions,  test  concepts  and  methodology,  tentative  evalua¬ 
tion  criteria,  and  test  profiles.  Further  development  and 
refinement  of  these  OT&E  objectives,  test  methodologies,  and 
evaluation  criteria  occur  during  actual  Test  Plan  develop¬ 
ment. 

•  Statement  of  Intended  Operational  Employment 

•  Contractor's  Statement  of  Work  (SOW) 

For  an  ATD  FOT&E,  at  a  minimum  the  following  additional  docu¬ 
mentation  should  exist  for  a  program  and  should  be 
referenced: 

•  IOT&E  or  QOT&E  Documentation  including  plans  and  reports, 
if  available 

•  Production  Contract  and  Statement  of  Work  (SOW) 

•  Using  command  ATD  operational/maintenance  concepts 
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COORDINATE 
WITH  EXISTING 
PLANNING 
GROUPS 


As  noted  earlier,  advance  planning  for  ATD  OT&E 
begins  as  early  as  possible  after  a  program  has 
been  identified;  i.e.,  either  formally  through 
receipt  of  a  PMD,  or  informally  through  inter¬ 
faces  with  the  procuring  organization.  The  advance  planning 
process  for  an  ATD  OT&E  may  have  included  inputs  from  a 
number  of  organizational  elements.  In  AFTEC,  for  example, 
the  management  of  advance  planning  is  the  responsibility  of 
Plans  (XR)  personnel.  Inputs  to  evaluation  approaches  for 


ATD  operational  effectiveness  will  have  been  generated  by 
Operations  Analysis  (OA),  and  for  operational  suitability  by 
Logistics  (LG)  elements.  Working  together,  these  elements 
and  others  produce  initial  concepts  for  the  ATD  evaluation 
approach  and  develop  preliminary  resource  estimates  for 
carrying  out  the  test  in  the  form  of  a  Test  Program  Outline 
(TPO).  In  addition,  a  preliminary  OT&E  program  schedule  will 
have  been  prepared  showing  anticipated  test  start  dates  and 
other  key  program  milestones. 


In  coordinating  with  the  existing  test  planning  groups  and 
reviewing  the  preplanning  to  date,  anticipated  test 
resources,  and  program  schedule,  the  test  director  should 
take  into  account  a  number  of  considerations.  First,  it  must 
be  remembered  that  much  of  the  advance  planning  that  may  have 
occurred  up  to  this  point  can  be  at  a  formative  stage  of 
development.  This  means  that  involvement  of  the  test  direc¬ 
tor  can  be  critical  to  the  constructive  refinement  and  devel¬ 
opment  of  test  plans.  Second,  estimates  of  anticipated  test 
resource  requirements  made  up  to  this  point  will  have  likely 
been  based  upon  experience  gained  in  earlier  similar  pro¬ 
grams.  They  will  have  been  made  of  necessity  at  an  earlier 
stage  to  allow  for  budget  submissions,  manpower  planning,  and 
lead  time  requirements  for  AFTEC,  MAJCOMS,  and  other  organi¬ 
zations  that  may  have  some  involvement  in  the  OT&E  (e.g., 
AFLC,  AFHRL,  and  ATC) .  It  should  be  expected  that,  as  test 
planning  continues,  the  TPO  will  be  periodically  updated/ 
revised  to  reflect  the  actual  requirements  of  the  test  plan. 
Third,  with  regard  to  program  schedule,  it  can  be  expected 
that  delays  in  the  program  may  occur.  Experience  has  shown 
that  aircrew  training  devices  can  be  expected  to  be  delayed 
in  schedule  from  three  to  twelve  months  or  more. 
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In  addition  to  coordinating  with  the  immediate 
test  planning  groups,  it  is  necessary  also  to 
coordinate  with  the  other  organizations  and  agen¬ 
cies  that  will  be  involved  in  the  OT&E.  During 
advance  planning,  a  certain  amount  of  precoordination  and 
preplanning  will  have  occurred.  At  this  point,  however,  it 
is  essential  for  the  test  director  to  determine  the  key  per¬ 
sonnel  in  those  other  organizations  and  establish  a  working 
liaison.  A  number  of  organizations  should  be  contacted 
including  the  SimSPO,  AFLC,  MAJCOM,  and  ATC.  The  SimSPO  will 
have  specific  information  as  to  program  schedule,  contractor 
contacts,  and  DTSE/Qual  test  plans.  The  AFLC  will  be 
involved  in  planning  the  test  of  device  suitability  factors. 
The  MAJCOM  (using  command}  which  will  be  providing  personnel 
resources  (i.e.,  aircrews  and  maintenance  personnel)  for  the 
test  should  be  contacted  as  early  as  possible  and  kept  up-to- 
date  on  test  resource  requirements.  The  ATC  will  be  able  to 
provide  information  regarding  planned  or  in-being  training 
programs. 


The  OTSE  test  director  should  now  be  up  to  date 
as  to  the  current  status  of  the  ATD  T&E  program 
and  aware  of  the  general  test  objectives  to  be 
accomplished.  His  next  task  is  that  of  estab¬ 
lishing  the  necessary  test  team.  A  test  program  team  must  be 
established  within  the  responsible  test  agency  (MAJCOM  or 
AFTEC)  to  assist  the  test  director  in  preparing  the  test 
plan.  Depending  upon  a  number  of  factors  (e.g.,  the  size  of 
the  program) ,  a  preliminary  test  program  team  may  al ready 
exist  and  may  even  have  made  inputs  for  test  resources  in  an 
initial  TPO.  Otherwise,  the  OT&E  test  director  will  need 
assistance  from  appropriate  personnel  to  plan  the  test. 
Inputs  to  evaluation  approaches  for  ATD  operational  effec¬ 
tiveness  may  be  obtained  from  AFTEC/MAJCOM  Operations 
Analysis  (OA)  or  AFHRL  personnel,  and  those  for  ATD  opera¬ 
tional  suitability  from  AFTEC/MAJCOM  Logistics  (LG)  person¬ 
nel  . 
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DEVELOP/REFINE 
TEST  DESIGN 
AND  METHODS 


Once  the  foregoing  preparatory  tasks  have  been 
carried  out,  it  is  time  to  address  the  details  of 
test  planning.  Required  activities  will  include 
development  and  refinement  of  specific  test 
objectives  and  subobjectives,  measures  of  effectiveness 
(MOEs),  and  evaluation  criteria. 


The  operational  test  and  evaluation  of  an  ATD  has 
two  principal  objectives.  The  first  of  these  is 
to  evaluate  the  operational  effectiveness  of  the 
device  including,  among  others,  assessments  of 
fidelity,  training  capability,  and  instructional  utility. 
The  second  overall  objective  is  to  evaluate  the  operational 
suitabil ity  of  the  device.  It  is  evident  that  these  objec¬ 
tives  have  meaning  in  a  more  general  sense;  they  are  not, 
however,  adequate  in  themselves  for  detailed  test  planning 
purposes.  What  must  be  done  is  to  refine  further  and  break 
down  the  global  objectives  into  concrete  and  manageable 
subobjectives.  The  operational  suitability  subobjectives  are 
well  defined  and  include  assessments  of  device  reliability, 
maintainabil ity,  avail abil ity,  and  logistics  supportabil ity . 
ATD  OT&E  subobjectives  in  the  area  of  operational  effec¬ 
tiveness,  however,  are  somewhat  more  difficult  to  generate. 
The  test  director  may  refer  to  past  ATD  OT&E  test  plans  as 
one  source  of  ideas  in  developing  subobjectives.  Caution 
should  be  exercised,  however,  when  reviewing  old  test  plans, 
as  they  may  contain  methodological  errors  and  be  otherwise 
unsuitable  to  the  current  program.  (See  AFM  55-43,  Vol .  II, 
Ch.  8-5,  Operational  Effectiveness:  Formulation  of  Objec¬ 
tives  and  Detailed  Planning.) 

The  operational  suitability  subobjectives  are  usually  better 
defined  and  include  specific  criteria  for  assessments  of 
device  reliability,  maintainability,  availability,  and 
logistics  supportabil  ity .  Again,  however,  the  test  director 
should  exercise  care  in  seeing  that  all  necessary  subobjec¬ 
tives  have  been  specified. 


GENERATE 
OBJECTIVES  AND 
SUBOBJECTIVES 


FOT&E  objectives  and  subobjectives  are  of  particular  concern 
since  the  general  purpose  of  an  FOT&E  is  to  provide  infor¬ 
mation  needed  to  integrate  the  ATD  effectively  within  its 


i ntended  trai ni ng  envi ro  nt.  The  identification  of  defi¬ 
ciencies  and  ver  i  ficatTo;,  of  correction  of  previously  iden¬ 
tified  deficiencies  is  also  an  important  part  of  the  purpose 
of  FOT&E. 

AFM  55-43  distinguishes  two  phases  of  FOT&E  (AFM  55-43,  Vol  . 
I,  Ch.  4-9).  Phase  I  FOT&E  is  done  to  refine  and  expand 
assessments  made  during  IOT&E  or  QOT&E,  and  it  tests  changes 
or  modifications  made  to  correct  deficiencies.  (AFTEC 
manages  Phase  I  for  major  and  designated  nonmajor  programs.) 
Phase  II  FOT&E  is  generally  conducted  by  the  using  command 
(and  monitored  by  AFTEC).  Phase  II  is  done  to  provide  infor¬ 
mation  needed  to  implement  the  ATD  into  its  intended  training 
environment.  For  a  given  ATD,  a  Phase  I  FOT&E  may  have  been 
made  unnecessary  (1)  by  the  on-site  phase  of  the  initial 
OT&E,  or  (2)  incorporation  of  the  Phase  I  FOT&E  objective  of 
checking  that  deficiencies  have  been  corrected  into  a  Phase 
II  FOT&E. 

Since  the  ATD  FOT&E  will  always  be  conducted  within  the  using 
command  environment,  more  extensive  and  longer  term  evalua¬ 
tions  can  be  accomplished.  Accordingly,  the  test  objectives 
appropriate  for  FOT&E  may  well  be  more  extensive  and  of  wider 
scope  than  those  of  a  I/QOT&E.  For  example,  a  "full  mission" 
ATD  installed  at  an  operational  base  may  have  unique  capabi¬ 
lities  to  simulate  selected  threat  environments.  A  meaning¬ 
ful  FOT&E  objective  in  this  case,  then,  might  be  to  determine 
the  capabilities  of  the  ATD  for  training  threat  counter¬ 
measures  and  combat  tactics  development.  This  determination 
likely  would  require  considerably  more  in  the  way  of  time  and 
aircrew  resources  than  would  be  appropriate  during  initial  or 
qualification  OT&E,  e.g.,  measures  of  effectiveness  would 
have  to  be  more  complex  than  the  simple  ratings  of  fidelity 
that  might  be  used  in  a  QOT&E. 


Measures  of  effectiveness  (MOE)  refer  to  the 
information  needed  to  support  the  specific  test 
subobjectives.  For  example,  if  one  of  the  test 
subobjectives  is  to  assess  the  fidelity  of  simu¬ 
lation  for  the  training  of  low  level  navigation,  a  measure  of 
effectiveness  might  be  a  rating  of  the  fidelity  of  visual 
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cues  represented,  as  perceived  by  experienced  pilots  who  have 
performed  low  level  navigation  in  the  ATD.  Given  the  possi¬ 
bility  of  conducting  a  transfer  of  training  evaluation  in 
FOT&E,  a  more  objective  MOE  might  be  used,  such  as  the  number 
of  trials  or  amount  of  time  required  to  reach  criterion  level 
on  low  level  navigation  tasks  in  the  airplane  following 
training  in  the  ATD. 


Once  specific  test  objectives,  subobjectives,  and 
MOEs  have  oeen  established,  it  is  necessary  to 
determine  appropriate  evaluation  criteria,  i.e., 
specified  values  and  standards  that  can  aid  the 
decision  making  process.  For  OT&E  data,  AFTEC  has  defined 
three  levels  of  evaluation  criteria:  thresholds,  standards, 
and  goals.  A  threshol d  is  the  minimum  level  of  acceptable 
performance  or  capabil  ity .  A  standard  is  that  level  of  per¬ 
formance  or  capability  that  will  satisfy  the  operational 
requirements  as  contained  in  the  system  operational  and  main¬ 
tenance  concepts  and  other  appropriate  documentation.  A  goal 
is  a  level  of  performance  or  capability  that  enhances  the 
system,  i.e.,  a  level  of  performance  superior  to  that  defined 
by  the  system  operational  concept.  Since  the  criteria  that 
would  be  appropriate  for  one  type  of  ATD  within  a  specific 
context  of  operational  training  utilization  may  differ  widely 
from  the  criteria  appropriate  for  another  device  and  training 
environment,  selection  of  evaluation  criteria  must  be  speci¬ 
fically  tailored  to  each  ATD  OT&E.  For  example,  a  device 
that  is  adequate  for  training  in  the  UPT  context  may  be  quite 
inadequate  for  the  continuation  training  context. 


Once  test  objectives  and  subobjectives  have  been 
defined  and  their  associated  evaluation  proce¬ 
dures  developed,  it  is  necessary  to  assess  the 
adequacy  of  resource  requirements  to  support  the 
test.  This  involves  updating  and  refining  the  test  program 
outline  (TPO)  to  assure  that  the  various  organizations  and 
agencies  to  be  involved  in  the  OT&E  are  willing  and  able  to 
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provide  the  necessary  support.  The  availability  of  user's 
resources  can  impact  greatly  the  overall  test  design  relative 
to  its  capability  to  accomplish  the  defined  objectives.  (See 
AFM  55-43,  Ch.  7-3,  Test  Program  Outline.) 
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OURCE  Concurrently  with  refining  the  TPO  it  is 

REMENTS  necessary  to  coordinate  resources  among  organiza- 
_ J  tions  for  supporting  the  test.  This  is  a  repeti¬ 
tive  process  wherein  resource  estimates  and 
requirements  are  updated  as  commitments  for  support  are  made 
more  and  more  concrete.  Once  appropriate  resource  coor¬ 
dinations  have  been  completed,  the  test  planner  should  obtain 
associated  Memoranda  of  Agreement  (MOA)  for  those  resources 
from  the  agencies  concerned.  In  most  cases,  MAJCOM  response 
to  the  TPO  will  serve  the  purpose  of  MOAs  or  MOUs  (Memoranda 
of  Understanding).  As  part  of  this  coordination  activity, 
procedures  for  reporting  Test  Discrepancies  and  Service 
Reports  (TD/SR)  should  be  developed  in  accordance  with  the 
guidelines  contained  in  T0-00-35D-54.  (See  AFM  55-43,  Ch . 
/-I,  Resource  Management.) 
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tUATE?  /  In  Process  °f  coordinating  test  resources  for 
personnel  and  materiel  requirements,  decisions 
will  be  required  as  to  whether  the  available 
resources  are  adequate  to  support  the  test  as  it 
is  planned.  These  are  especially  important  decisions,  since 
a  "well  planned"  test  is  worth  little  if  it  cannot  be  sup¬ 
ported  adequately.  Because  ATD  OT&E  may  involve  testing  at 
the  contractor's  facility  or  on-site  at  the  using  command's 
facility,  coordination  as  appropriate  for  each  is  required. 
If  the  test  director  determines  that  some  element(s)  of  the 
test  cannot  be  supported,  it  may  be  necessary  to  revise  and 
otherwise  modify  a  particular  subobjective  or  change  the 
evaluation  procedure  to  accommodate  the  subobjective  with  the 
available  resources.  In  any  case,  it  will  be  necessary  to 
revise/modify  some  portion  of  earlier  planning. 


Prior  to  writing  the  OT&E  test  plan,  the  test 
director  must  review  the  test  approach  with 
working  level  participants  in  the  test  planning 
effort  to  obtain  inputs  regarding  test  resources 
and  the  general  assignment  of  responsibilities  to  accomplish 
the  test  program.  At  this  review,  any  problems  or  potential 
problems  related  to  test  resource  commitment  and  test  sche¬ 
duling  should  be  addressed. 
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A  draft  of  the  test  plan  should  now  be  written  in 
accordance  with  the  format  established  by  AFM 
55-43  or  the  appropriate  MAJCOM  OT&E  planning 
guidance  documentation.  Annexes  to  the  test  plan 
directed  to  the  technician  level  may  be  added  or  deleted,  as 
required,  for  communicating  the  methodologies  and  detailed 
procedures  to  be  used.  Volume  II,  Annex  8-8,  of  AFM  55-43 
contains  the  general  format  and  content  requirements  for  OT&E 
Test  Plans.  In  preparing  this  draft  of  the  test  plan  it  will 
not  be  necessary  to  have  completed  the  development  of  all 
procedures  that  will  be  employed  in  testing  and  evaluating 
the  operational  effectiveness  or  operational  suitability 
objectives.  A  representative  sample  of  the  procedures  to  be 
employed  should,  however,  be  included  as  an  annex  to  the 
plan.  These  might  include,  among  others,  the  rating  scales 
to  be  used  in  fidelity  and  training  capability  assessments,  a 
list  of  ATD  training  tasks  to  which  those  scales  are  to  be 
applied,  and  the  methods  for  analyzing  and  interpreting  the 


The  Draft  Test  Plan  must  now  be  reviewed  and 
coordinated  within  AFTEC  and  with  the  MAJCOMs  and 
other  interested  agencies.  Several  reviews  and 
meetings  may  be  needed  to  finalize  the  test  plan 
and  obtain  the  required  coordination  of  test  resources. 
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Based  upon  inputs  received  during  review  of  the 
Draft  Test  Plan,  a  final  version  of  the  Test  Plan 
is  prepared  reflecting  the  most  current  stage  of 
pi anning . 


Prior  to  publishing  the  Test  Plan  and  its  distri¬ 
bution  to  the  relevant  ATD  OT&E  participants,  it 
is  reviewed  and  approved  in  accord  with  estab¬ 
lished  admini strative  procedures  (e.g.,  USAFTAWCR 
55-8,  Ch .  5C).  This  review  and  approval  process  should 

address  all  aspects  of  the  plan  to  assure  that  it  meets  tech¬ 
nical  and  quality  standards.  It  will  include  review  of  test 
objectives,  evaluation  approach,  and  any  unresolved  issues 
that  may  exist  at  this  time.  The  review  process  may  often 
include  briefings  by  the  test  director  for  various  levels  of 
management  within  and  outside  his  organi zation .  The  factors 
mentioned  above  may  be  covered  in  these  briefings,  as  well  as 
issues  relating  to  test  resource  coordination  among  organiza¬ 
tions.  For  example,  in  the  case  of  combined  DT&E/IOT&E  or 
Qual/QOT&E  programs,  the  Test  Plan  should  take  into  account 
the  data  gathered  by  the  SimSPO  and  its  appl  icabil  ity  to 
operational  effectiveness/suitability  objectives.  Also, 
there  may  be  a  sharing  of  resources  during  a  combined  eval¬ 
uation  that  should  be  taken  into  account  both  from  a  budget 
standpoint  and  from  the  standpoint  of  the  quantity  of 
resources  available  for  the  time  in  question.  Based  upon 
review  of  the  completed  Test  Plan,  the  appropriate  AFTEC  or 
MAJCOM  commander  may  recommend  modifications  and/or  refine¬ 
ments  prior  to  publishing  and  distributing  the  Test  Plan. 
This  may  necessitate  changes  to  test  objectives,  evaluation 
approach,  distribution  of  responsibility,  or  other  aspects  of 
the  pi  an . 


PUBLISH 

TEST 

PLAN  Following  review  and  approval  ,  the  OT&E  Test  Plan 

is  published  and  distributed  to  participating 
commands  and  agencies.  Concurrently  with  the 
publishing  of  the  Test  Plan,  detailed  test  proce¬ 
dures  are  further  developed  and  refined. 


/  PRE- 
PUBLICATION 
REVIEW  AND 
V  APPROVAL  J 


COMPLETE 
FINAL  DRAFT 
LEST  PLAN 


37 


FINALIZE  In  the  time  between  the  publishing  of  the  Test 

SUPPORT  PI  an  and  the  beginning  of  the  "Execution  Phase," 

AGREEMENTS  it  will  be  necessary  to  continue  updating  and 

-  finalizing  test  resource  agreements.  Often,  a 

substantial  period  of  time  may  pass,  and  priorities  within 
supporting  organizations  and  agencies  can  change.  It  is 
important,  therefore,  to  maintain  an  appropriate  level  of 
communication  and  coordination  among  test  elements.  Memo¬ 
randa  of  Agreement  (MOA)  should  be  continually  updated  and 
validated.  Also,  some  time  between  the  planning  and  execu¬ 
tion  phases,  the  test  director  must  plan  for  housing  and 
transportation  of  his  personnel  to  and  from  the  test  site. 


END 

PLANNING  PHASE 


) 


c 


BEGIN 
EXECUTION  PHASE 


VERIFY 

TEST 

SUPPORT 


EXECUTION  PHASE 


Concurrently  with  activation  of  the  test  team,  it 
is  critical  to  verify  that  negotiated  test 
resources  (aircrews,  maintenance  personnel,  test 
equipment,  etc.)  are  available  and  ready  to  sup¬ 
port  the  test.  Test  support  problems  identified  during  this 
verification  must  be  resolved  by  the  test  director  as  early 
as  possible  to  avoid  adverse  impacts  on  the  test  schedule. 
Also,  prior  to  commitment  of  resources  the  test  director 
should  determine  the  progress  of  the  contractor  and  his 
readiness  for  an  operational  test.  Often,  the  contractor  is 
not  prepared  for  evaluation  other  than  his  own  acceptance 
test  procedures.  If  possible,  a  "ready  for  test"  checklist 
should  be  used  to  assure  that  device  engineering  development 
is  complete  and  device  availability  figures  are  such  that  an 
"operational"  training  configuration  exists. 
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Prior  to  the  start  of  actual  data  collection  on 
the  ATD,  provisions  should  be  made  for  the 
training  of  test  personnel  (maintenance  personnel 
and  aircrews)  in  the  operation  and  use  of  the 
device.  This  training  should  include  both  familiarization 
with  device  basic  operation  as  well  as  the  use  of  instruc¬ 
tional  features.  Training  should  include  a  certain  amount  of 
hands-on  practice  to  minimize  confusion  in  the  operation  of 
the  device  during  the  subsequent  test. 


Once  the  preceding  activities  have  been  accom¬ 
plished,  execution  of  the  test  plan  begins.  The 
test  director  and  his  deputies  are  responsible 
for  the  overall  management  of  the  test  and  the 
associated  data  collection.  The  test  director  should  ensure 
that  the  detailed,  day-to-day  test  procedures  are  being 
carried  out  to  support  the  objectives  of  the  test  plan.  For 
the  combined  DT&E/IOT&E  or  Qual/QOT&E  programs,  this  should 
be  done  in  close  coordination  with  the  Qual  Test  Director. 
As  noted  earlier,  during  the  in-plant  OT&E,  emphasis  is  upon 
assessing  ATD  fidelity,  training  capability,  and  ATD  reliabi¬ 
lity  (during  the  reliability  demonstration).  The  on-site 
phase  of  OT&E  will  be  concerned  with  maintainability  assess¬ 
ment,  during  which  time  the  system  would  not  be  available  for 
training  capability  since  faults  would  be  inserted  that  would 
interrupt  continuous  training  use  of  the  ATD.  In  some  cases, 
however,  a  schedule  can  be  developed  hereby  suitability  data 
collection  may  occur  in  shifts  that  are  not  required  for 
effectiveness  testing,  e.g.,  aircrews  0800-1600,  M-demo 
1700-0100,  etc.  (See  AFM  55-43,  Ch .  9,  Test  Execution.) 

During  the  in-plant  phase  of  the  OT&E,  device  maintenance 
actions  will  typically  be  handled  by  the  contractor.  During 
the  on-site  phase  of  testing,  i.e.,  the  mai ntainabil ity 
demonstration,  device  maintenance  actions  will  be  carried  out 
by  the  appropriate  Air  Force  test  team  personnel.  Therefore, 
provision  should  be  made  for  any  maintenance  personnel 
training  that  may  be  necessary  for  the  on-site  phase. 


PERFORM 

TESTS 


Data  collected  in-plant  will  include  device 
fidelity,  training  capability,  IOS  capability 
(operational  effectiveness),  and  the  ATD  relia¬ 
bility  and  software  suitability  assessments.  As 
previously  discussed,  the  data  coming  out  of  an  on-site  ATD 
OT&E  can  be  directed  to  a  wide  range  of  objectives  depending 
upon  the  information  required  to  integrate  the  ATD  into  its 
intended  operating  environment.  These  data  may  include  addi¬ 
tional  ATD  operational  suitability  assessments  as  well  as  the 
more  substantive  operational  effectiveness  determinations. 


PREPARE 

SERVICE 

REPORTS 


WjQE  One  of  the  primary  functions  of  an  OT&E  is  to 

jqrTs  detect  and  report  deficiencies  that  may  have  an 

_ adverse  impact  on  the  operational  effectiveness 

or  suitability  of  a  system.  The  Service 
Reporting  guidelines  and  procedures  contained  in  T0-00-35D-54 
and  AFM  55-43,  Chapter  10,  should  be  reviewed  and  followed  in 
this  area. 


SUBMIT 

STATUS 

REPORTS 


PORTS  Status  reports  provide  periodic  updates  and 

important  test  findings  to  interested  Air  Force 
and  MAJCOM  agencies  before  the  operational  test 
program  is  complete  and  the  final  report  is 
submitted.  They  are  usually  submitted  by  letter  or  message, 
but  their  format  and  content  may  be  adjusted  to  meet  indivi¬ 
dual  test  program  requirements.  The  OT&E  test  plan  should 
specify  the  format  and  frequency  for  these  status  reports. 
Volume  II  of  AFM  55-43,  Annex  11-2,  contains  guidelines  for 
preparing  status  reports. 


PRODUCE 

INTERIM 

REPORTS 


An  interim  report  is  more  formal  than  a  status 
report.  It  is  generally  prepared  only  upon  spe¬ 
cific  MAJCOM  or  HQ  USAF  request  to  support  a 
major  program  milestone  or  to  report  problems  in 
"testing.  It  should  provide  a  summary  of  the  test  results, 
from  the  beginning  of  the  test  to  date.  Generally,  this 
coordinated  report  should  cover  each  specific  test  objective 
in  the  test  plan.  (AFM  55-43  recommends  that  unless  an 

interim  report  is  absolutely  necessary,  it  should  be  avoided, 
because  erroneous  conclusions  can  be  drawn  from  insufficient 
data  and  incomplete  testing.) 


The  test  director  will  normally  give  periodic 
test  program  status  briefings  to  his  management 
throughout  the  active  test  period.  Briefings  are 
presented  to  a  series  of  audiences  including 
immediate  management,  HQ  USAF,  AFTEC,  SimSPO,  MAJCOMs,  and 
other  organizations  and/or  agencies  that  have  some  interest 
or  involvement  in  the  test  program  ( e -g . ,  AFHRL) .  (AFM  55-43 
contains  guidelines  for  briefing  content  in  Volume  I,  Ch. 
11-13.) 


PREPARE 
AND  GIVE 
BRIEFINGS 


Q 


END 

EXECUTION  PHASE 


0 


Q 


BEGIN 
REPORTING  PHASE 


D 


(REVIEW  DATA 
AND  REPORTS 
TO  DATE 


REPORTING  PHASE 


A  number  of  inputs  covering  various  test  objec¬ 
tives  will  have  been  prepared  by  the  test  team 
members  in  the  form  of  status  reports,  progress 
reports,  and  other  initial  draft  material.  The 
first  activity  in  the  reporting  phase,  then,  is  to  review 
those  initial  drafts  and  associated  supporting  data  for 
accuracy  and  readability. 
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PRODUCE 

DRAFT 

REPORT 


In  conjunction  with  the  above  activity,  the  test 
director  prepares  a  Draft  Report  of  the  ATD  OT&E, 
as  appropriate.  This  report  covers  the  test 
results,  conclusions,  and  recommendations.  This 
document,  when  finalized,  serves  two  important  purposes. 
First,  it  disseminates  the  test  information  required  by  deci¬ 
sion  makers,  planners,  and  operators,  and,  second,  it  pro¬ 
vides  a  formal,  permanent  document  of  the  test  results  for 
the  Air  Force. 


AFM  55-43,  Volume  II,  Annex  11-1,  contains  specific  guidance 
for  preparation  of  this  report.  The  reporting  format 
described  in  AFM  55-43  is  intended  to  be  appropriate  for  both 
AFTEC-  and  MAJCOM -man aged  programs;  however,  specific 
guidance  may  also  exist  within  the  MAJCOM  for  report  format 
and  content  that  should  be  followed.  Reference  to  earlier 
ATD  OTSE  reports  may  provide  additional  insights  and  guideli¬ 
nes  for  formatting  the  report.  With  regard  to  the  technical 
content  of  the  report  in  key  areas  of  interest,  the 
appropriate  volumes  of  this  Handbook  provide  specific 
guidance. 


The  test  director  schedules  a  review  of  the  final 
draft  of  the  ATD  OT&E  report  within  HQ  AFTEC 
and/or  the  MAJCOM,  as  appropriate.  Several 
cycles  of  review  and  modification  may  be  needed 
to  finalize  the  report. 


f  \ 

CONDUCT 

REVIEW 

V  J 


Based  upon  inputs  received  during  the  review  a 
Draft  Final  Report  is  prepared  for  command 
approval . 
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PUBLISH  I  Following  review  of  the  completed  Final  Report, 

REPORT  I  the  AFTEC  or  MAJCOM  commander,  as  appropriate, 

may  approve  its  publication.  However,  he  may 
recommend  that  further  modifications  and/or 
refinements  be  made  prior  to  publishing  and  distributing  the 
report. 


CLOSE  OUT 

TEST 

PROGRAM 

remaining 

END  > 

REPORTING  PHASE J 

The  test  director  closes  out  the  case  file  in 
accord  with  appropriate  MAJCOM  or  AFTEC  proce¬ 
dures.  This  will  include  storing/ fil ing  data, 
completing  financial  actions,  and  disposition  of 
remaining  test  equipment,  among  other  activities. 


CHAPTER  4 


ATD  ACQUISITION/T&E  EVENT  FLOW 


INTRODUCTION 

The  preceding  two  chapters  have  described  the  ATD  acquisition  and 
OT&E  processes  largely  as  unrelated  and  somewhat  time  independent 
activities,  even  though  these  two  processes  are,  in  reality,  closely 
interrelated  and  highly  time  dependent.  They  were  presented  as 
separate  entities  mainly  to  allow  a  more  straightforward  explication 
of  each  process.  It  is,  however,  important  that  the  test  director 
have  a  clear  understanding  of  the  temporal  relationship  between  those 
ATD  acquisition  events  described  in  Chapter  2  and  the  Air  Force  test 
and  evaluation  activities  enumerated  in  Chapter  3.  The  purpose  of 
this  chapter,  therefore,  is  to  show  that  relationship. 

The  relationship  between  the  acquisition  and  T&E  processes  is 
also  affected  by  the  "size"  of  the  ATD  procurement  of  interest.  ATD 
procurements  may  range  in  complexity,  for  example,  from  programs  as 
big  as  the  B-52  WST  to  some  as  small  as  a  single  CPT.  Understandably, 
variations  in  size  and  complexity  of  ATD  acquisition  programs  have  an 
impact  on  ATD  OT&E  planning  and  execution.  This  chapter  is  also 
intended  to  illustrate  that  effect. 


ATD  ACQUISITION  AND  OT&E  INTERFACE 

The  temporal  interface  between  the  ATD  acquisition  process  and 
ATD  OT&E  planning  and  execution  is  shown  in  Figure  4-1.  Those  blocks 
above  the  time-line  depict  the  more  significant  acquisition  events; 
those  below  the  time-line  represent  the  major  OT&E  activities.  From 
Figure  4-1  it  can  be  seen  that  both  acquisition  and  T&E  activities 
begin  with  the  issuance  of  a  Program  Management  Directive  (PMD).  This 
event  marks  Air  Force  approval  of  a  MAJCOM  Mission  Element  Needs 
Statement  (MENS)  and  signals  the  beginning  of  both  advanced  procure¬ 
ment  and  test  and  evaluation  planning.  Obviously,  only  a  relatively 
low  level  of  advanced  OT&E  planning  is  required  up  to  the  award  of  a 
development/production  contract  to  one  or  more  vendors. 

Contract  award  marks  the  start  of  the  Production  Phase  which 
includes  preliminary  and  detailed  design  of  the  ATD  and  its  fabrica¬ 
tion,  Detailed  ATD  OT&E  planning  begins  during  the  production  phase 
and  normally  intensifies  around  the  time  of  critical  design  review 
(CDR).  At  the  end  of  the  production  phase,  the  ATD  is  declared  "Ready 
for  Test"  (RFT).  All  OT&E  planning  and  preparatory  activity  must  have 
been  completed  by  the  time  the  device  becomes  RFT,  so  that  necessary 
test  and  evaluation  activities  can  proceed  as  smoothly  as  possible. 
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TIME  (MONTHS)  ACQUISITION 
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Detailed  OT&E 


Form  TPWG 

1 

Prepare  TEMP  j 

Acquire  and 

Review  OT&E 
Documents 

Develop  Initial 

Concepts  For 

OT&E  Approach 

Prepare  PMP 

— 

Review  Existing 
Program  Documents 

Develop 

Preliminary  OT&E 

Program  Schedule 

Estimate  OT&E 

Resources  Required  1 

--Prepare  TPO  1 

Coordinate  with 

Ex  1  sting  PI ann 1 ng 
Groups 

other  Test 
E  I  aments _ 

Establish  Test 
Program  Team 


Figure  4-1.  ATD  acquisition  and  TuE  flow. 


Start  Air  Force 
Test  and  Evaluation 
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Concurrent  or  Combined  Testing 


As  can  be  seen  from  Figure  4-1,  a  portion  of  the  OT&E  effort 
occurs  after  the  ATD  has  been  installed  at  its  operational  location. 
The  early  OT&E  (IOTSE  or  Q0T4E)  which  occurs  at  the  vendor's  plant, 
normally  occurs  concurrently  with  development  of  qualification  testing 
( DT &E  or  QT&E).  The  on-site  QT&E  may  or  may  not  occur  concurrently 
with  development  testing.  Concurrent  OTSE  and  DT&E/QT&E  are  generally 
referred  to  as  "Combined  Testing."  Testing  activities  are  combined 
whenever  separate  testing  activities  would  be  less  efficient  in  the 
use  of  available  testing  time  or  support  resources. 

Combined  DT&E/IOT&Z  activities  begin  as  soon  as  possible  after 
the  device  is  determined  to  be  ready  for  testing.  Some  DT&E/QT&E  spe¬ 
cific  events  usually  ^ccu r  first  because  they  are  critical  technical 
and  engineering  tests  that  must  be  performed  to  clear  the  system  for 
further  testing.  However,  even  during  this  very  early  DT&E/QT&E  test 
period,  0T8E  personnel  participate  as  observers  to  become  familiar 
with  the  system  and  to  gain  access  to  any  test  data  that  may  support 
the  OT&E  effort. 

Following  these  front-end  DT&E/QT&E  events  there  are  a  number  of 
joint  DT&E/QT&E  and  IOT&E/QOT&E  test  events.  An  excellent  example  of 
a  shared  test  event  is  the  DT&E/QT&E  Reliability  Demonstration  during 
which  OT&E  personnel  may  also  be  performing  specific  operational 
effectiveness  evaluations  such  as  device  functional  fidelity  assess¬ 
ments  and/or  trainability  estimates.  As  much  as  possible  of  the  in- 
plant  operational  effectiveness  evaluation  usually  takes  place  during 
the  Reliability  Demonstration.  Not  all  OT&E  objectives  can  or  will  be 
satisfied  by  shared  test  events.  As  a  result,  there  will  be  a  few  T&E 
activities  that  will  occur  either  before  or  after  combined  testing. 
Close  coordination  with  the  SimSPO  program  manager  and  the  contractor 
is  required  to  assure  that  all  necessary  such  test  events  are  properly 
schedul ed. 

In-plant  combined  T£E  continues  until  sufficient  data  have  been 
collected  to  support  a  conditional  acceptance  of  the  ATD  being  tested. 
Following  this  in-plant  testing  and  any  required  deficiency  correc¬ 
tions,  the  ATD  is  disassembled  and  shipped  to  where  it  will  be  used 
operationally.  Disassembly,  shipment,  and  reassembly  may  take  from 
2-6  months.  After  the  ATD  is  reassembled  at  its  on-site  location. 


*It  will  be  recalled  that  the  key  to  understanding  the  distinc¬ 
tion  between  DT&E/IOT&E  and  QT&E/QOT&E  is  the  source  of  funding  for 
such  tests.  Testing  supported  by  RDT&E  funds  is  called  DT&E/IOT&E; 
and  testing  funded  by  O&M  dollars  is  referred  to  as  QTSE/QOT&E. 
Otherwise,  DT4E/I0T4E  and  QT4E/Q0T&E  procedures  are  basically  the 
same. 
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combined  testing  resumes  on-site  and  continues  until  the  Air  Force 
acceptance  is  effected  by  DO-25 0  sign-off. 

Post  DD-250  Sign-off  Testing 

As  Figure  4-1  shows,  I0T&E/Q0T&E  usually  continue  for  some  time 
following  Air  Force  acceptance  until  all  remaining  test  objectives 
have  been  satisfied.  Whereas  the  OT&E  test  director  shared  the  device 
with  SimSPO  personnel  during  combined  testing,  he  now  must  share  the 
device  with  MAJCOM  user  personnel.  These  people  are  anxious  to  become 
operational  and,  as  a  consequence,  the  test  director  may  expect  some 
problems  regarding  the  availability  of  device  time  and  its  controlled 
utilization  for  test  purposes. 

Completion  of  IOT&E/QOT&E  marks  the  end  of  OT&E  for  many  ATDs. 
However,  it  is  possible  that  additional  follow-on  operational  test 
evaluations  { FOT&E)  may  be  required.  As  indicated  in  Figure  4-1, 
these  may  occur  at  any  time  during  the  life-cycle  of  the  ATD.  Such 
FOT&Es  as  are  required  usually  address  some  specific  area  of  device 
effectiveness  or  suitability  concern.  A  modification  to  a  device's 
visual  system,  for  example,  might  generate  an  interest  in  a  training 
transfer  evaluation  of  that  capability  for  additional  syllabus  cover¬ 
age.  Another  example  of  an  FOT&E  requirement  might  result  from 
post-IOT&E  software  suitability  concerns  or  a  need  for  updated 
supportabil  ity  information. 


INFLUENCE  OF  PROCUREMENT  COMPLEXITY 

Although,  as  previously  pointed  out,  the  basic  procedures 
involved  in  DT&E/IOT&E  or  QT&E/QOT&E  are  pretty  much  the  same,  there 
are  a  few  differences  in  the  time  sequencing  of  key  events  and  the 
implications  drawn  from  the  test  results  for  subsequent  ATD  procure¬ 
ment  action.  The  major  factor  influencing  the  time  sequencing  of  T&E 
activities  is  the  complexity  of  the  procurement  itself.  Figures  4-2 
and  4-3  are  provided  to  illustrate  the  nature  of  T&E  event  sequencing 
for  procurements  of  differing  levels  of  complexity.  Figure  4-2  shows 
the  T&E  event  flow  for  an  ATD  procurement  involving  some  RDT&E  monies, 
and  Figure  4-3  shows  the  T&E  event  flow  for  a  procurement  not  involv¬ 
ing  RDT&E  funds.  Figure  4-2  also  is  representative  of  a  one-of-a-kind 
device  (e.g.,  the  Aerial  Refueling  Part-Task  Trainer  or  the  Simulator 
for  Air-to-Air  Combat),  while  Figure  4-3  is  illustrative  of  a  larger- 
scale  multiple  device  procurement  (e.g.,  the  B-52  WST). 

One-of-a-Kind  Developmental  Procurements 

A  number  of  ATD  procurements  involve  untried  technology  applica¬ 
tions  or  are  for  devices  for  which  training  transfer  effecti veness  may 
not  previously  have  been  demonstrated .  Although  decision  may  later  be 
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made  to  procure  additional  devices  of  the  type  being  tested,  the 
device  initially  is  considered  a  one-of-a-kind  developmental  device, 
being  procured  from  a  single  vendor. 

As  can  be  seen  from  Figure  4-2,  a  portion  of  IOT&E  may  be  con¬ 
ducted  before  the  first  full  production  decision  using  prototypes, 
preproduction  items,  or  pilot  production  items.  The  primary  purpose 
of  that  part  of  IOT&E  is  to  give  decision  makers  an  estimate  of  the 
probable  operational  effectiveness  and  operational  suitability  of  an 
ATD  before  the  final  production  go-ahead  decision  is  made.  In  this 
way  any  uncertainties  regarding  the  ultimate  operational  effectiveness/ 
suitability  of  the  ATD  (such  as  would  perhaps  be  present  if  the  ATD 
were  being  designed  to  exploit  a  new  visual  or  motion  cueing  technol¬ 
ogy)  can  be  resolved  prior  to  a  major  expenditure. 

Large  Scale,  Multiple  ATD  Procurements 

OT&E  as  shown  in  Figure  4-3  is  conducted  on  large  scale  systems 
mainly  to  verify  that  the  ATD  will  perform  to  specification  and  that 
it  should  meet  user  effectiveness  expectations.  This  type  of  OT&E  is 
also  used  during  "fly-before- buy"  competitive  procurements  such  as  the 
B -52  WST.  Of  particular  significance  in  this  situation  is  the  "Quick- 
look"  OT&E  report  (see  Figure  4-3).  That  report  is  based  on  compa¬ 
rable  T&E  (QT&E  and  QOT&E)  activities  performed  in  each  vendor’s 
facility.  The  results  provided  in  that  report  are  based  on  a  number 
of  specific  and  possibly  selection  sensitive  dedicated  OT&E  tests  that 
will  have  occurred.  These  tests  must  be  carefully  evaluated  and 
objectively  summarized  in  that  brief  OT&E  report  for  use  in  support  of 
the  source  selection  process. 

After  the  source  selection  decision  has  been  made,  the  winning 
contractor  updates  and  refurbishes  the  ATD  as  is  necessary  prior  to 
renewed  OT&E.  Additional  in-plant  QT&E/QOT&E  is  then  performed,  as 
required,  on  the  winning  device  before  it  is  disassembled  and  shipped 
to  its  operational  location. 

Reliability  and  maintainability  demonstrations  for  the  winning 
device  can  be  conducted  either  during  the  completion  of  in-plant 
testing  or  after  the  device  has  been  installed  and  checked  out  at  its 
on-site  location. 

Once  the  ATD  has  been  installed  on-site,  additional  OT&E  is 
accomplished  as  necessary.  OT&E/QOT&E  ends  with  formal  acceptance  of 
the  ATD  by  the  Air  Force  (signified  by  signing  of  the  DD-250).  Fol¬ 
lowing  Air  Force  acceptance  of  the  ATD,  further  OT&E  activities  may 
continue  for  as  long  as  10-12  months.  Such  activities  may  be  limited 
to  "clean-up,"  or  they  may  involve  one  or  more  transfer-of- training 
studies. 
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It  is  important  that  the  test  director  be  prepared  for  the  admin¬ 
istrative  and  management  pressures  characteristic  of  this  type  of  ATD 
OT&E.  The  major  pressure  usually  comes  from  those  deadlines  which  are 
tied  to  the  source  selection  process.  Both  the  contractors  and  the 
SimSPO  have  a  great  deal  of  inte  ^st  in  those  OT&E  events  which  will 
contribute  to  the  ultimate  source  selection.  As  a  result,  .  _•  test 
director  often  is  pressed  to  meet  these  source  selection  deadlines, 
and  must  compress  and/or  streamline  some  of  his  presource  selection 
Q0T6E  activities. 
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APPENDIX  A 


GLOSSARY 

This  glossary  contains  terminology  and  acronyms  related  to  air¬ 
crew  training  device  design,  utilization,  and  testing.  Many  of  the 
terms  and  definitions  provided  here  relate  to  the  technology  of  ATD 
design  and  development  generally,  rather  than  to  OT&E  specifically. 
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ATD:  Aircrew  Training  Device.  Refers  to  training  media  which  provide 
for  some  form  of  active  trainee  practice  through  simulation  of 
essential  task  characteristics.  These  training  media  include 
cockpit  familiarization  and  procedures  trainers,  operational 
flight  trainers,  part-task  trainers,  instrument  flight  trainers, 
and  v^apon  system  trainers. 

ATP:  Acceptance  Test  Procedures.  Tests  conducted  to  demonstrate  that 
ATD  engineering  design  is  complete,  that  design  and  production 
risks  are  minimized,  and  that  the  device  fulfills  the  specifica¬ 
tions  of  the  procuring  contract. 

AUGMENTATION:  Providing  information  which  does  not  exist  in  the  real 
world,  or  an  enhancement  of  naturally  occurring  information. 

AUTO-DEMO:  Automated  Demonstration.  An  instructional  feature  that 

permits  the  standardized  presentation  of  a  mission  segment  or 
entire  simulated  flight.  All  cues  of  consequence,  including  the 
visual  scene  (if  present),  motion  cues,  primary  flight  controls 
and  displays,  crew  communications  and  sensor  displays,  are  repro¬ 
duced  for  the  selected  mission  segment  or  maneuver  through  com¬ 
puter  control . 

AUTOMATED  CONTROLLERS:  Computer-based  systems  that  use  mathematical 
models  to  determine  controller  messages  and  issue  controller 
messages  using  computer  generated  speech. 

AUTOMATED  PERFORMANCE  ALERTS:  Signals  intended  to  enhance  the  moni¬ 
toring  of  student  performance  by  combining  the  capabilities  of 
automated  performance  measurement  with  automated  alerting  signal 
generation  for  instructors  and/or  students. 

AUTOMATED  PERFORMANCE  MEASUREMENT  (APM) :  The  computer-based  applica¬ 
tion  of  technology  to  monitoring,  recording,  processing  and 
displaying  objective,  quantitative  information  that  describes 
student  performance  and  assists  in  diagnosing  student  learning 
problems. 

AVAILABILITY:  Availability  reflects  the  readiness  of  the  ATD  to  per¬ 
form  its  training  mission. 

AVERAGE  DEVIATION:  A  measure  of  variability  that  describes  the  extent 
to  which,  on  the  average,  individual  scores  depart  from  the  mean. 
Average  deviation  is  the  sum  of  the  absolute  differences  of  indi¬ 
vidual  scores  from  the  mean,  divided  by  the  total  number  of 
scores. 

BAFO:  Best  and  Final  Offer 
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BITE:  Built-in  Test  Equipment. 


CAMERA/MODEL  SYSTEM:  A  type  of  simulator  visual  system  which  con¬ 
sists  of  a  scale  model  of  terrain,  aircraft,  or  other  features, 
and  is  viewed  by  a  closed  circuit  television  camera. 

CCT:  Combat  Crew  Training. 

CCTV:  Closed  Circuit  Television. 

CDR:  Critical  Design  Review.  A  premanufacture  review  of  detailed 
device  design. 

COMPUTER  IMAGE  GENERATION  (CIG):  Creation  of  synthetic  visual  images 
by  computer  processing  of  a  numerical  data  base  containing  infor¬ 
mation  about  the  objects  and  features  which  are  part  of  a 
displayed  visual  scene. 

CPT:  Cockpit  Procedures  Trainer. 

CT:  Continuation  Training:  Training  conducted  routinely  in  opera¬ 
tional  squadrons,  or  proficiency  training  conducted  periodically. 

CUE:  In  this  Handbook,  cue  means  some  critical  feature  or  stimuli 
which  gives  important  information  to  a  pilot  or  other  aircrew 
member. 

DCAS:  Defense  Contract  Administrative  Services. 

DELAY:  The  difference  in  time  between  when  a  change  occurs  in  a  simu¬ 
lated  visual  scene,  movement  of  the  cockpit,  or  response  of  a 
force  cueing  device  and  when  it  should  have  occurred. 

DISPLAY  CHANNEL:  An  independent  ATD  visual  system  display.  Multiple 
display  channels  often  are  used  to  create  large  field  of  view 
visual  systems. 

DT&E:  Development  Test  and  Evaluation. 

ECP:  Engineering  Change  Proposal  . 

FIDELITY:  The  degree  of  correspondence  between  a  simulated  aircrew 

station  and  environment  and  the  actual  aircrew  station  and 
environment.  Also  see:  physical  fidelity;  psychol ogical  fidel¬ 
ity;  and  realism. 
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FIELD  OF  VIEW:  The  dimensions  of  the  area  of  a  visual  display  which 
can  be  seen.  Expressed  in  terms  of  visual  angle. 

FLIGHT  CHARACTERISTICS  FIDELITY:  The  extent  to  which  aircraft  control 
and  response  characteristics  are  reproduced  accurately  in  an  ATD. 

FORCE  CUEING  DEVICES:  ATD  mechanizations  that  are  intended  to  provide 
onset  and  sustained  motion-related  flight  cues.  They  include: 
G-s eats;  G-suits;  seat  shakers;  helmet  loaders;  arm  loaders;  and 
visual  system  greyout/bl ackout  capabilities. 

FOTAE:  Follow-on  Operational  Test  and  Evaluation. 

FREEZE:  An  instructional  support  feature  that  allows  simulator  param¬ 
eters  to  be  fixed  at  values  existing  v^ten  freeze  is  actuated. 

HARDCOPY:  An  ATD  instructional  feature  that  enables  the  Instructor/ 

Operator  to  reproduce  on  paper  data  displayed  at  the  IOS. 

HUMAN  FACTORS:  The  application  of  information  about  human  capabili¬ 
ties  and  limitations  to  the  design,  use,  and  evaluation  of  machi¬ 
nes,  systems,  and  devices  of  all  kinds. 

ILITIES:  See  Operational  Suitability. 

IMAGE  The  picture  or  scene  created  by  a  simulator  visual  system 
which  is  viewed  by  a  pilot  or  other  aircrew  member. 

IMAGE  QUALITY:  Characteristics  of  the  appearance  of  an  image,  inde¬ 
pendent  of  the  scene  content  of  the  image. 

INITIALIZATION:  Initialization  involves  specifying,  usually  from  the 

instructor/ operator  console,  the  parameters  of  interest  and  their 
values  for  positioning  and  configurino  an  ATD  within  a  gaming 
area. 

IN-PLANT:  At  the  contractor's  faci’it^ 

INSTRUCTIONAL  SUPPORT  FEATURE:  ATD  capabilities  that  allow  the 

instructor  to  manipulate,  supplement,  and  otherwise  control 
trainee  learning  experiences  to  increase  the  rate  of  learning  and 
to  maximize  the  level  of  skill  achieved. 

INSTRUCTOR  CONSOLE:  See  Instructor/Operator  Station  (IOS). 

INSTRUCTOR/OPERATOR  STATION  (IOS):  The  aircrew  training  device  man- 
machine  interface  where  active  control  and  monitoring  of  training 
events  occurs. 
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INTERVAL  SCALE:  A  rating  scale  that  utilizes  name  or  number  cate¬ 
gories,  implies  order  among  categories,  and  approximates  equal 
intervals  between  items. 

IOT&E:  Initial  Operational  Test  and  Evaluation. 

IRON  PILOTS:  Computer  controlled  adversaries,  typically  used  in  air 
combat  maneuvering  training  in  ATDs. 

ISD:  Instructional  system  development:  Procedural  approaches  to  the 
analysis  of  training  requirements  and  the  development  of  training 
programs  and  systems. 

LOGISTICS  SUPPORTABILITY:  Logistics  supportabil ity  includes  a  number 
of  concerns,  each  of  which  relates  to  the  support  and/or  mainte¬ 
nance  of  the  ATD.  These  concerns  include  supply,  test  support 
equipment  (SE),  technical  data,  facilities,  personnel  and  train¬ 
ing,  and  transportation  and  handling. 

LSET:  Logistics  Supportabil ity  Evaluation  Team. 

MAINTAINABILITY:  A  characteri Stic  of  system  design  and  installation  V 

that  affects  the  ease  or  difficulty  with  which  that  system  may  be 
retained  in,  or  restored  to,  a  specific  serviceable  condition. 

MAJOR  PROGRAM:  An  acquisition  program  involving  more  than  $75  million 
RDT&E  or  $300  million  production  costs. 

MEAN:  The  average  of  a  set  of  scores;  i.e.,  the  sum  of  all  scores 

divided  by  the  total  number  of  scores. 

MEDIAN:  The  middle  score  of  a  set  of  rank-ordered  scores.  For  an  odd 
number  of  scores  the  median  is  simply  the  middle  score;  for  an 
even  number  of  scores  the  median  is  the  average  of  the  two  middle 
scores. 

MOA:  Memorandum  of  Agreement. 

MOCK-UP  REVIEW:  A  review  by  the  Air  Force  of  a  nonworking  model  of 
the  device  to  be  built.  Mock-up  review  is  usually  held  con¬ 
currently  with  PDR. 

MODE:  The  score  that  occurs  most  often.  Sometimes  a  set  of  scores 

will  be  multi-modal,  i.e.,  two  or  more  scores  occur  equally 
often . 

MOE:  Measure  of  Effectiveness. 
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NOMINAL  SCALE:  A  rating  scale  that  utilizes  name  categories,  without 
suggesting  any  numerical  relationship  among  those  categories. 

NONMAJOR  PROGRAM:  An  acquisition  program  involving  less  than  $75 
million  RDT&E  or  $300  million  production  costs. 

O&M:  Operations  and  Maintenance. 

OFT:  Operational  Flight  Trainer. 

ON-SITE:  At  the  user's  facility. 

OPERATIONAL  DEFICIENCY:  A  deficiency  in  the  design  or  performance 
characteristics  of  an  ATD  in  terms  of  its  operability  and  train¬ 
ing  utility. 

OPERATIONAL  EFFECTIVENESS:  How  well  the  device  fulfills  its  intended 
mission  in  its  intended  environment.  For  ATDs,  operational 
effectiveness  means  training  utility. 

OPERATIONAL  SUITABILITY:  Operational  suitability  factors  pertain  to 
how  well  the  device  meets  accepted  equipment  serviceability 
requirements  within  its  intended  operating  and  maintenance 
environment.  These  factors  include  hardware  reliability,  main¬ 
tainability,  availability,  logistics  supportabil ity ,  and  software 
suitabil  ity. 

ORDINAL  SCALE:  A  rating  scale  that  utilizes  name  or  number  categories 
and  which  implies  rank  order  among  those  categories,  but  does  not 
concern  the  size(s)  of  the  interval  between  items. 

OT&E:  Operational  Test  and  Evaluation. 

OTEMP:  OT&E  Master  Program. 

PDR:  Preliminary  Design  Review. 

PERCEPTION:  The  acquisition  of  information  about  the  world  through 

the  human  senses. 

PHYSICAL  FIDELITY:  The  degree  to  which  an  ATD  replicates  precisely 
the  aircraft  it  represents. 

PLATFORM  MOTION  SYSTEMS:  ATD  mechanizations  that  provide  typically 
from  3  to  6  degrees  of  freedom  of  ATD  cockpit  movement.  Flight 
motion  simulated  may  include  pitch,  roll,  yaw,  heave  (vertical 
movement),  front/back  and  left/right  (lateral  movement). 

PMD:  Program  Management  Directive. 
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PMP:  Program  Management  Plan. 

PRACTICE:  Repetition  of  task  performance  to  improve  or  maintain  pro¬ 
ficiency  on  that  task. 

PROGRAMMED  MISSION  SCENARIOS:  Highly  structured  sets  of  events  that 
are  caused  to  occur  automatically,  under  computer  control . 

PSYCHOLOGICAL  FIDELITY:  The  degree  of  correspondence  of  cues  and 

responses  accompanying  task  performance  in  an  ATD  to  those 
characteristics  of  analogous  performance  in  an  aircraft. 

QOT&E:  Qualification  Operational  Test  and  Evaluation. 

QTP:  Qualification  Test  Procedures:  See  ATP. 

RATING  SCALE:  A  data  collection  tool  that  enables  a  person  to  express 
an  estimate,  judgment,  or  opinion  about  some  quality  or  quantity 
in  terms  of  a  category  or  number. 

RATIO  SCALE:  A  rating  scale  that  possesses  all  the  properties  of  an 
interval  scale,  plus  has  an  absolute  zero  point. 

REALISM:  The  extent  to  which  an  aircrew  member's  experiences  in  an 

ATD  correspond  to  experiences  as  they  actually  would  occur  in  the 
aircraft  under  a  given  set  of  conditions.  Also  see  physical 
fidel  ity . 

RECORD/REPLAY:  The  instructional  feature  that  provides  the  capability 
in  an  ATD  to  record  relevant  system  parameters  and  then  use  these 
data  to  present  student  pilot  performance  in  a  review  mode  in  the 
ATD. 

RELIABILITY:  The  likelihood  that  the  ATD,  and  its  major  subsystems 

will  work  satisfactorily  when  needed  for  training.  Reliability 
is  expressed  in  terms  of  the  frequency  with  which  failures  occur 
(e.g..  Mean  Time  Between  Critical  Failure,  Mean  Time  Between 
Maintenance  Action). 

REMOTE  DISPLAY:  An  ATD  instructional  feature  that  permits  alpha¬ 

numeric  and  graphic  data  on  an  IOS  display  to  be  displayed  simul¬ 
taneously  at  the  trainee  station. 

RESOLUTION:  The  smallest  separation  between  two  objects  in  a  display 
that  can  be  detected,  usually  by  the  human  eye. 

RESPONSE:  Any  motor,  perceptual  or  mental  act  by  a  person,  generally 
refers  to  an  element  of  an  overall  action  as  opposed  to  the 
overall  action  itself. 
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RETENTION:  The  capacity  to  remember  task  performance  requirements 

after  a  period  of  time  when  the  task  has  not  been  practiced. 

RDTSE:  Research  and  Development  Test  and  Evaluation. 

RFP:  Request  for  Proposal . 

RTA:  Responsible  Test  Agency. 

SAT:  Software  Analysis  Team. 

SimSPO:  Simulator  System  Program  Office. 

SIOE:  Statement  of  Intended  Operational  Employment. 

SOFTWARE:  Computer  Programs. 

SON:  Statement  of  Operational  Need. 

SOW:  Statement  of  Work. 

SPECIFICATION:  A  contractual  statement  describing  the  device  to  be 

built  in  terms  of  its  functions  and  characteristics. 

SPO:  System  Program  Office. 

STANDARD  DEVIATION:  A  measure  of  variability  of  individual  scores 
about  the  mean.  The  standard  deviation  is  the  square  root  of  the 
sum  of  squared  deviations  about  the  mean  divided  by  the  total 
number  of  scores. 

STORE/RESET:  An  ATD  instructional  feature  that  permits  the  simulation 
to  be  returned  or  reset  to  a  set  of  conditions  that  existed  at  an 
earlier  point  in  time.  This  feature  is  often  used  for  repetitive 
practice  of  training  maneuvers. 

T&E:  Test  and  Evaluation. 

TD:  Test  Director. 

TEMP:  Test  and  Evaluation  Master  Plan. 

TEST  PLAN:  The  formal  document  that  contains  the  test  objectives, 
methods,  and  the  resources  required  to  conduct,  analyze,  and 
report  the  test. 

TM:  Test  Manager. 

TO:  Technical  Order. 


60 


TPO:  Test  Program  Outline. 

TPWG:  Test  Planning  Working  Group. 

TRAINING  EFFECTIVENESS:  The  demonstrated  improvement  in  trainee  per¬ 
formance  following  practice  in  the  ATD.  Often  expressed  in  terms 
of  the  transfer  effectiveness  ratio  (TER). 

TRAINING  EFFICIENCY:  The  extent  to  which  available  resources  (includ¬ 
ing  time)  are  used  economically  during  training. 

TRAINING  OBJECTIVES:  Explicit  statements  of  the  goals  of  training 

including  tasks  to  be  performed,  the  performance  standards  for 
each  task,  and  the  conditions  under  which  those  tasks  are  to  be 
performed. 

TRAINING  REQUIREMENTS:  General  statements  of  task  performance  skills 
required  for  operational  proficiency.  Also,  general  statements 
of  performance  skills  that  require  periodic  practice  in  order  to 
maintain  proficiency. 

TRAINING  SCENARIO:  A  predefined  sequence  of  training  events  used  to 
exercise  the  capabilities  of  an  ATD  in  a  specific  area  of 
intended  training  usage. 

TRANSFER  OF  TRAINING:  The  transfer  of  skills  learned  in  one  context 
(e.g.,  an  ATD )  to  a  different  context  (e.g.,  an  aircraft).  The 
carry-forward  of  trained  performance  to  real  world  applications. 

TRANSITION  TRAINING:  Training  for  aircrew  members  transitioning  to 
different  operational  aircraft. 

VIRTUAL  IMAGE:  In  visual  simulation,  a  virtual  image  appears  to  be  at 
a  greater  distance  (e.g.,  optical  infinity)  than  the  actual 
display  surface. 

WST:  Weapon  System  Trainer. 
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APPENDIX  B 


ANNOTATED  BIBLIOGRAPHY* 

This  annotated  bibliography  contains  summaries  of  selected  docu¬ 
ments,  regulations,  and  other  publications  relevant  to  ATD  operational 
test  and  evaluation.  As  a  guide  to  secondary  sources  for  technical 
information,  this  bibliography  is  formatted  to  facilitate  access  and 
retrieval  of  needed  materials. 


^Portions  of  this  bibliography  (designated  *)  were  excerpted  from 
Spears,  Sheppard,  Roush,  and  Richetti,  Simulator  training  requirements 
and  effectiveness  study  (STRES):  Abstract  bibliography  ( AFHRL-TR-80- 
38 J .  Brooks  AfB,  TX:  Air  Force  Human  Resources  Laboratory,  January 
1981. 
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ADMINISTRATIVE/ MANAGEMENT  DOCUMENTS 


Department  of  the  Air  Force.  Test  and  Evaluation  (AFR  80-14). 

Washington,  X:  Headquarters  US  Air  Force,  July  1976. 

This  regulation  defines  "policy  and  procedure  for  managing  test 
and  evaluation  activities  during  the  development,  production  and 
deployment  of  defense  systems  in  the  Air  Force."  It  establishes 
the  management  relationships  among  the  implementing  command,  the 
Air  Force  Test  and  Evaluation  Center,  and  the  operating  and  sup¬ 
porting  commands  in  successive  phases  of  a  system's  life  cycle, 
from  the  conceptual  phase  through  deployment  and  employment.  It 
includes  discussion  of  the  various  types  of  test  and  evaluation, 
test  documentation,  assignment  of  responsibilities,  administra¬ 
tion,  and  a  glossary  of  terms  relating  to  test  and  evaluation. 
AFR  80-14  applies  to  all  Air  Force  organi zati  ons . 


Department  of  the  Air  Force.  Test  and  Evaluation  (MAC  Regulation 
55-80).  Scott  AFB,  IL:  Headquarters  Military  Airlift  Command, 

December  1976. 

This  regulation  establishes  "policies  and  outlines  procedures  for 
managing,  conducting,  and  supporting  programs  for  test  and  evalu¬ 
ation  of  new  or  improved  systems  and  equipment  within  MAC."  It 
contains  brief  definitions  of  the  various  types  of  test  and  eval¬ 
uation,  and  assigns  admini strati ve  and  fiscal  responsibilities 
among  organizations  who  participate  in  T&E.  Included  are  general 
formats  for  use  by  agencies  and  organizations  requesting  test 
programs,  preparing  test  plan  outlines,  test  orders,  test  plans, 
and  final  reports.  A  brief  glossary  is  provided  that  defines 
terms  relating  to  test  and  evaluation.  MACR  55-80  applies  to  all 
MAC  organizations  and  agencies  with  regard  to  test  and  evaluation 


Department  of  Air  Force.  Management  of  Operational  Test  and  [valua¬ 
tion  (AFM  55-43  ).  Washington,  X:  Headquarters  U.  S.  Air  Force,  June 
1979  (Vol .  I),  July  1979  (Vol  .  II). 

In  two  volumes,  AFM  55-43  is  designed  to  explain  the  operational 
test  and  evaluation  (OT&E)  program,  and  how  it  relates  to  other 
Air  Force  and  DoD  activities.  In  summarizes  the  principles  and 
procedures  that  will  promote  consistent  OT&E  management  through¬ 
out  the  Air  Force.  It  establishes  guidelines  for  standardi zing 
the  planning,  conducting,  and  reporting  of  OT&E  programs  in  the 
Air  Force;  however,  because  the  scope  of  those  programs 


varies,  judgment  must  be  used  In  applying  these  guidelines  to 
each  individual  program.  The  major  commands  may  set  specific 
command  policies  and  procedures  not  only  to  implement  the  manual, 
but  to  provide  for  specific  procedures  and  tests  outside  its 
scope.  The  two  volumes  of  AFM  55-43  correspond  topically,  but 
differ  in  terms  of  the  scope  of  their  coverage.  Volume  I  is 
directed  to  "all  levels  of  management  and  presents  the  background 
and  underlying  philosophy  for  current  0T4E  policies  and  proce¬ 
dures.  It  provides  the  general  guidelines  on  planning,  managing, 
conducting,  and  reporting  on  0T4Es."  The  intent  of  Volume  II  is 
more  to  provide  the  how-to  information  and  is  directed  to  the 
"doers"  (i.e.,  the  test  planner,  test  director,  and  test  team 
member).  It  contains  a  wealth  of  specific  guidance,  techiques, 
and  procedures  relevant  to  OT&E.  This  is  a  must-have  source 
document  for  the  OT&E  test  manager/director .  The  content  of  AFM 
55-43  volumes  is  shown  below  (chapters  are  same  for  both 
volumes) . 
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Introduction 

Chapter 

2. 
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Chapter 
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Chapter 

4. 

Types  of  Operational  Test  and  Evaluation 

Chapter 

5. 

Objectives  of  Operational  Test  and  Evaluation 

Chapter 

6. 
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Acquisition  Process 
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Test  Execution 
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Chapter  11. 

Test  Reporting 

Department  of  the 

Air 

Force.  Test  and  Evaluation:  A  Guide  for  Test 

and  Evaluation  Management  (ASDP  80-14).  Wright-Patterson  AFB,  OH: 
Headquarters,  Aeronautical  Systems  Division  (AFSC),  October  1978. 


This  document  provides  guidance  for  management  of  test  and  evalu¬ 
ation  (TSE)  during  the  acquisition  phase  of  weapon  systems  or 


equipment.  The  purpose  of  this  document  is  to  assist  program 
managers  in  focusing  on  those  essential  actions  that  must  be 
accomplished  as  key  events  during  both  major  and  nonmajor  acqui¬ 
sition  programs.  This  guidance  is  oriented  primarily  toward  T&E 
as  regulated  by  APR  80-14;  however,  the  concepts  and  procedures 
may  be  tailored  for  application  to  any  Air  Force  T&E  program. 
The  information  in  this  document  emphasizes  the  required  T&E 
planning  and  program  documentation,  and  it  is  organized  to  serve 
middle  management  such  as  test  managers  and  project  officers.  A 
second  thrust  is  to  identify  actions  or  situations  that  require 
special  attention  because  they  illuminate  pitfalls  or  successful 
experiences,  a  knowledge  of  which  will  aid  test  managers  destined 
to  encounter  similar  situations  in  the  system  acquisition  pro¬ 
cess. 


Department  of  the  Air  Force.  Logistics  Assessment  (AFTECP  400-1). 
Kirtland  AFB,  NM:  Headquarters  Air  Force  Test  and  Evaluation  Center, 
May  1978. 

This  document  is  a  guide  for  logistics  program  managers  of  Air 
Force  Test  and  Evaluation  Center  (AFTEC)  who  participate  in  plan¬ 
ning,  execution,  and  reporting  of  operational  test  and  evaluation 
of  systems  during  the  acquisition  process.  Although  this  docu¬ 
ment  is  intended  for  personnel  assigned  to  the  AFTEC  Directorate 
of  Logistics,  portions  may  also  be  useful  to  other  Air  Force 
activities  involved  in  operational  test  and  evaluation  of  equip¬ 
ment.  Included  are  discussions  of  the  acquisition  process,  com¬ 
puterized  maintenance  and  logistics  data  systems,  test  planning, 
execution,  and  reporting.  Appendices  contain  information  on 
abbreviations,  definitions,  computerized  products,  and  publica¬ 
tions  that  are  applicable  to  operational  test  and  evaluation. 


Air  Force  Test  and  Evaluation  Center  (AFTEC).  Cost  of  Ownership 
Handbook.  Kirtland  AFB,  W:  Author,  May  1976. 

This  procedure  handbook  was  prepared  for  use  in  assessing  the 
cost  of  ownership  (COO)  during  system  test  efforts.  It  defines 
cost  of  ownership  and  each  of  the  cost  elements  that  comprise  the 
term.  In  addition,  it  describes  a  methodology,  equations,  and 
techniques  that  may  be  used  for  the  development  of  cost  of  owner¬ 
ship  reporting  requirements.  A  rather  detailed  description  is 
included  of  AFTEC 's  life  cycle  cost  of  ownership  model.  This 
handbook  is  heavily  oriented  toward  aircraft  systems,  although 
cost  elements  are  identified  for  a  number  of  other  systems 
including  airborne  missiles,  communications,  electronic,  and 
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meteorological  systems,  IC8M  systems,  drone  systems,  and  simula¬ 
tors. 


Department  of  the  Air  Force.  Operational  Test  and  Evaluation  Management 
Procedures  (USAFTAWCR  55-8).  Eglin  AFB,  FL:  USAF  Tactical  Air  Warfare 
Center  (TAC),  January  1980. 

This  regulation  "establishes  operational  test  and  evaluation 
(OT&E)  policies  and  guidance,  and  presents  procedures  and 
[defines!  responsibilities  for  the  accomplishment  of  USAFTAWC's 
OT&E  mission."  Like  AFM  55-43,  this  document  contains  a  wealth 
of  information  relevant  to  system  OT&E.  The  range  of  coverage  is 
broad,  from  general  OT&E  policy  and  guidance  to  specific  instruc¬ 
tions  and  directives  for  project  planning,  execution,  and 
reporting.  A  number  of  useful  tables  and  figures  are  provided 
including,  among  others,  a  sample  test  program  schedule,  project 
plan,  and  project  report.  This  document  was  written  to  minimize 
the  need  for  extensive  reference  to  other  source  documents. 
Consequently,  there  is  a  large  measure  of  redundancy  to  AFM 
55-43.  Nevertheless,  this  is  a  must-have  document  for  the  TAC 
test  manager/director  of  an  ATD  OT&E. 


TECHNICAL  DOCUMENTS 


Bryan,  G.  L.,  &  Regan,  J.  J.  Chapter  1 7 :  Training  system  design.  In 
H.  P.  Van  Cott  &  R.  G.  Kinkade  (Eds  ,  Human  engineering  guide  to 
equipment  design.  Washington,  DC:  U  ..  Government  Hnntinq- Officer 
T972t  (Pp.  633-666)  * 

The  purpose  was  to  present  a  conceptual  design  of  training  based 
on  educational  technology  and  system  design  techniques.  Design 
of  devices  must  be  concerned  specifically  with  the  configuration 
and  capabilities  of  instructor  and  student  stations.  The 
instructor  must  be  able  to  exercise  system  functions  and  capabil¬ 
ities  in  a  timely  and  effective  manner,  and  the  student  should 
receive  the  guidance,  feedback,  etc.,  in  an  efficient  and  effec¬ 
tive  way.  (An  earlier  warning  was  repeated  regarding  realism  for 
the  sake  of  realism  "since  the  student  station  is  the  place  where 
costly  and  unnecessary  realism  is  apt  to  show  up  in  a  simulator." 
The  instructor's  station  must  also  provide  selected  information 
about  students'  performance,  and  automatic  data  acquisition 
should  be  considered  to  aid  in  interpreting  performance.  "In 
short,  the  designer  should  repeatedly  ask  himself  what  the 
instructor  must  be  able  to  do  to  assist  the  student  maximally, 
and  then  arrange  for  the  instructor  to  be  able  to  do  those  things 
with  the  machine  and  not  in  spite  of  it."  Once  training  equip¬ 
ment  is  designed,  a  utilization  guide  should  be  prepared.  It 
should  establish  procedures  for  distribution  of  practice  and  rest 
periods,  pacing  task  presentations,  ensuring  "overlearning"  as 
needed,  preparing  for  practice,  and  incorporati ng  mental  prac¬ 
tice.  It  should  also  describe  special  methods  of  use  such  as 
demonstrations  and  joint  practice  by  two  or  more  trainees. 
Trade-offs  regarding  preprogrammed  (canned)  training  and  flexible 
adaptations  of  content  should  be  approached  carefully,  though 
with  computers  (and  presumably,  modules)  preprogrammed  content 
can  be  used  with  considerable  flexibility.  Provisions  should  be 
made  to  ensure  user  acceptance  of  training  devices  and  related 
training  systems,  which  requires  carefully  planned  communication 
between  designers  and  users.  Finally,  an  evaluation  of  the 
training  system  should  be  made  over  a  "reasonably  long  period  of 
time"  with  the  system  in  full  operation. 


Caro,  P.  W.,  Pohlmann,  L.  D. ,  &  Isley,  R.  N.  Development  of  simulator 
instructional  feature  design  gui  des  (Tech.  Rep".  TR  "75-17 ) .  —  Peri  sac  ol  a  , 
F L :  Seville  Research  Corporation7~0c tober  1979.  * 

There  is  no  "convenient  mechanism  to  acquaint  simulator  designers 
with  training  requirements  that  pertain  specifically  to 


instructional  features."  The  purpose  of  this  project  was  to 
develop  guides  for  defining  instructional  feature  (IF)  require¬ 
ments  that  could  be  used  for  user-designer  communication.  The 
approach  aimed  at  a  set  of  guides  appropriate  for  a  variety  of 
simulators,  using  a  particular  device  currently  under  development 
(an  F-16  simulator)  as  a  vehicle  for  developing  them.  Informa¬ 
tion  regarding  IFs  was  obtained  through  observations  and  reviews 
of  existing  simulators,  simulator  instruction  and  roles  of 
training  personnel  and  training  practices  and  requirements  for 
the  F-16.  Twelve  IFs  were  selected  for  exemplar  development: 
(a)  Record/Pi  ayback— permits  replay  of  a  preceding  recorded 
segment  of  device  operation;  (b)  Store/Reset  Current  Conditions — 
permits  device  to  be  reset  to  an  earlier  set  of  conditions;  (c) 
Remote  Display--for  displaying  alphanumeric  and  graphic  data  to 
the  learner;  { d)  Hardcopy— for  printouts  of  performance  records; 
(e)  Manual  Freeze-permits  simulated  actions  to  be  suspended  or 
"frozen";  (f)  Automatic  Freeze— simil ar  to  (e)  except  that  it 
occurs  automatically  under  specified  conditions;  (g)  Parameter 
Freeze — for  selective  freeze  of  only  one  or  a  few  parameters 
rather  than  all  simulator  operation;  (h)  Demonstrati  on— permits 
automated  task  performance  by  the  device  itself  to  illustrate  how 
tasks  may  be  performed,  and  the  conditions  and  effects;  (i) 
Demonstration  Preparation— permits  an  instructor  to  prepare  his 
own  Demonstration  for  future  use ;  (j)  Automatic  Malfunction 

Insertion — inserts  predetermined  simulated  malfunctions  automati¬ 
cally  under  specified  conditions;  and  (k)  Automatic  Malfunction 
Insertion  Exercise  Preparation— permits  an  instructor  to  prepare 
his  own  automatic  malfunction  exercises.  The  guides  for  each 
feature  were  presented  in  a  format  consisting  of  the  feature 
identification,  its  definition,  its  purpose  and  intended  use, 
concurrent  events  that  are  compatible/ incompatibl e  with  use  of 
the  feature,  and  a  flow  diagram  of  the  functions  involved  in  its 
use . 


Charles,  J.  P.  Instructor  pilot's  role  in  simulator  training  (Phase 
II)  (NAVTRAEQUIPCCto  /6-C-O034-1 ) .  Orlando,  Fl:  JTaval  Training 

"Equipment  Center,  August  1977.  * 

An  earlier  conducted  Phase  I  of  this  effort  found  that  "the  simu¬ 
lator  instructor  for  pilot  training  is  typically  untrained  for 
that  job,  is  not  provided  essential  information  for  the  task  .  . 
.  and  is  expected  to  perform  at  a  console  not  designed  for  the 
job."  Pursuant  to  overcoming  these  difficulties,  the  present 
Phase  II  study  determined  an  instructor  pilot's  (IP)  functions  in 
simulator  training.  IP  functions  are  grouped  under  ten  heads, 
each  including  subheads.  The  ten  simulator  IP  functions  were: 
(1)  Prepare;  (2)  Brief;  (3)  Initialize;  (4)  Train;  (5)  Evaluate; 


(6)  Debrief;  (7)  Manage  Data;  (8)  Develop  Syllabus;  (9)  Train  IP; 
and  (10)  Train  Self/Peer. 


Flexman,  R.  E.,  Roscoe,  S.  N.,  Williams,  A.  C.,  Jr.,  &  Williges,  B.  H. 
Studies  in  pilot  training:  The  anatomy  of  transfer.  Aviation  Research 
Monographs,  1972,  2^1). 

This  document  contains  a  discussion  of  the  measurement  of 
transfer,  including  the  concepts  of  a  Transfer  Effectiveness 
Ratio  (TER)  and  Incremental  Transfer  Effectiveness  Ratio  (ITER). 
It  reports  two  early  experimental  studies  on  the  transfer  effec¬ 
tiveness  of  flight  simulators  as  a  function  of  specific  contact 
and  instrument  flight  tasks.  A  number  of  worthwhile  insights  are 
provided  in  a  conclusion  section  that  relate  to  the  measurement 
of  transfer.  The  complexities  of  measuring  simulator  transfer 
effectiveness  in  the  context  of  a  total  training  curriculum  were 
noted,  and  the  need  was  stressed  for  more  comprehensive  research 
strategies  to  deal  with  those  complexities. 


Jeantheau,  G.  G.  Handbook  for  training  systems  evaluation  (NAVTRA- 
DEVCEN  66-C -0113-2) .  Orlando,  FL:  Naval  Training  Device  Center, 
January  1971.  * 

The  purpose  of  this  report  was  to  present  the  basic  notions 
underlying  training  device  effectiveness  in  the  context  of  prac¬ 
tical  constraints  on  research  in  training  device  settings.  Four 
levels  of  evaluation  were  discussed:  (1)  qualitative  assessment, 
(2)  noncomparative  measurement,  (3)  comparative  measurement,  and 
(4)  transfer  of  training.  The  first  level,  qualitative  assess¬ 
ment,  does  not  involve  measurement  of  any  kind,  but  is  based  on 
judgments  made  against  a  prior  criteria  of  cost  of  the  device, 
and  the  training  situation  that  research  and  experience  have 
shown  contribute  to  effectiveness.  The  second  level,  noncompara¬ 
tive  measurement,  is  the  crudest  form  of  quantitative  assessment. 
It  involves  a  measurement  of  training  performance  from  the 
beginning  to  the  end  of  training.  The  gain  scores  represent  the 
effectiveness  of  training  with  the  device.  Level  three,  compara¬ 
tive  measurement,  is  where  statistical  evidence  and  experimental 
control  come  into  play.  Comparisons  of  group  progress  using  dif¬ 
ferent  media  or  training  programs  are  the  bases  for  evaluating 
separate  media  and  programs.  To  ensure  comparabil ity  between 
situations,  control  must  be  exercised  over  the  training.  The 
fourth  level,  transfer  of  training,  requires  establishing  that 
training  in  the  simul ator  results  in  improved  performance  in  an 
operational  situation.  Each  succeeding  level  provides  increasing 


rigor  and  entails  increased  problems  of  coordination  and  coopera¬ 
tion  with  the  training  activity,  but  provides  the  investigator 
with  increasing  levels  of  validity  and  reliability.  Illustra¬ 
tions  and  blank  evaluation  forms  are  provided  to  help  clarify  the 
discussion.  Evaluation  techniques  are  placed  in  perspective 
relative  to  each  other  and  to  situational  constraints  that  affect 
their  utility. 


Mi  1 1 er ,  R .  B .  Handbook  on  training  and  training  equipment  design 
( WADC  Tech.  Rep.  53-1 36  V.  Wright-Patterson  AFB,  OH:  Wright  Air 
Development  Center,  June  1953.  * 

The  purpose  was  to  provide  a  handbook  of  principles  for  special¬ 
ists  in  training.  Theoretical  ,  empirical,  and  intuitive  grounds 
for  the  principles  were  explained  as  well.  As  a  handbook,  points 
are  presented  briefly  (although  clearly),  and  each  point  is 
itself  an  abstract  of  an  often  voluminous  research  literature. 
It  was  written  for  persons  who  already  have  more  than  an  intro¬ 
ductory  knowledge  of  the  psychology  of  learning.  (Nevertheless, 
persons  who  are  not  specialists  in  this  area  can  gain  consider¬ 
able  insight  into  training  issues  from  it.)  There  are  seven 
lengthy  sections  covering  topics  as  follows:  Section  1  provides 
an  overview  of  human  learning  emphasi zi ng  cues  and  mediational 
processes  (especially  verbal  mediation)  and  learning  conditions 
and  criteria  that  lead  to  cue  and  response  discriminations  and 
their  retention.  Section  2  defines  the  role  of  the  instructor 
as  he  must  motivate,  direct,  and  evaluate  students,  and  arrange 
the  learning  situation.  Section  3  addresses  the  role  of  demon¬ 
stration  (guidance)  in  training,  with  special  emphasis  on  the  use 
of  training  devices.  Section  4,  which  comprises  40  percent  of 
the  text  of  the  report,  goes  into  detail  regarding  the  use  of 
knowledge  of  resul ts  (feedback)  to  manage  and  control  student 
learning.  Section  5  addresses  problems  of  simulation,  device 
requirements.  Section  6  discusses  student  motivation  as  deter¬ 
mined  by  biographical  factors,  aspirations,  prestige,  etc.  Then, 
factors  emerging  during  training  that  affect  motivation  are 
explained.  Section  7  provides  a  schema  for  preparing  specifica¬ 
tions  for  a  training  device.  An  index  is  included  that  is  espe¬ 
cially  valuable  for  location  of  discussions  of  the  concepts. 
This  is  a  classic  document  on  training  and  the  use  of  training 
devices.  The  author  had  an  unusual  mastery  of  the  psychology  of 
learning,  and  he  could  state  it  in  a  straightforward  manner.  It 
would  be  well  for  persons  involved  in  simulator  programs  to 
return  to  this  report  for  guidance. 


t 


Semple,  C.  A.  Executive  summary:  A  report  of  the  simulator  training 
requirements  and  effectiveness  (STRES)  (AFHRL-TR-80-63) .  Brooks  AFB, 
TXT  Air  Force  Human  Resources  Laboratory,  1980. 

The  Simulator  Training  Requirements  and  Effectiveness  Study 
(STRES)  was  conceived  as  a  means  of  identifying  and  making 
available  the  existing  information  related  to  Air  Force  simulator 
training.  The  purpose  was  to  assemble  a  technical  data  base  to 
provide  guidance  for  the  enhancement  of  present  training,  as  well 
as  for  the  focus  of  R&D  needed  to  enhance  future  simulator-based 
training.  A  series  of  reports  was  produced  that  address  a  number 
of  relevant  issues,  i.e.,  device  fidelity,  instructional  support 
features,  device  utilization,  life  cycle  cost  and  worth  of  owner¬ 
ship.  In  addition,  areas  of  information  "gap"  were  identified 
and  research  plans  developed  to  address  those  areas.  Finally,  a 
comprehensi ve  abstract  bibliography  was  prepared  to  provide 
meaningful  background  information  to  users  and  to  serve  as  a 
guide  for  secondary  sources  of  technical  information.  In  addi¬ 
tion  to  the  “executive  summary"  referenced  above,  the  following 
six  reports  were  prepared: 

Semple,  C.  A.,  Hennessy,  R.  T. ,  Sanders,  M.  S.,  Cross,  B.  K. , 
Beith,  B.  H. ,  &  McCauley,  M.  E.  Aircrew  training  device 
fidelity  features  ( AFHRL-TR -80-36 ) .  ~'BrooJcs~BFffr" TX :  "  'Ai r 
Force  Human  Resources  Laboratory,  1980. 

Semple,  C.  A.,  Cotton,  J.  C.,  &  Sullivan,  D.  J.  Aircrew 
training  device  instructional  support  features  ( AFHRL-TR -80- 
5T8).  Brooks-  AFB,  TX:  Air  Force  Human  Resources  Laboratory, 
1980. 

Caro,  P.  W.,  Shelnutt,  J.  B.,  &  Spears,  W.  D.  Aircrew 
training  devices:  Utilization  ( AFHRL-TR-80-35) .  Brooks  AFB, 
TX:  Air  Force  Human  Resources  Laboratory,  1980. 

All  bee,  K.  E.,  X  Semple,  C.  A.  Aircrew  training  device  life 
cycle  cost  and  worth  of  ownership  ( AFHRL-TR-80-34 ) .  Brooks 
AFB,  TXT  Air  Force  Human  Resources  Laboratory,  1980. 

Prophet,  W.  W.,  Shelnutt,  J.  B.,  &  Spears,  W.  0.  F uture 
research  plans:  A  report  of  the  simulator  training  require¬ 
ments  "aho  "effect! venes¥~  study  (SIklST  (AF  HRl-1 R-BU-3/ ) . 
Brooks  AFB,  TX:  Air  Force  Human  Resources  Laboratory,  1980. 

Spears,  W.  D. ,  Sheppard,  H.  J.,  Roush,  M.  0.,  II,  &  Richetti, 
C.  L.  Abstract  bibliography:  A  report  of  the  simulator 
training  requirements  and  effectiveness  study  (STRES)  (AFHRL- 
TR-80-38 ) .  Brooks  AFB ,  TXT  ATr  Force  Human  Resources 
Laboratory,  1980. 


Wheaton,  G.  R.,  Rose,  A.  M.,  Fingerman,  P.  W.,  Korotkin,  A.  L.,  & 
Holding,  D.  H.  Evaluation  of  the  effectiveness  of  training  devices: 
Literature  review  and  preliminary  model  [Research  Memorandum  /b-b). 
Arlington,  VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  April  1976. 

The  purpose  of  the  report  was  to  develop  the  foundations  for,  and 
present  a  preliminary  statement  of,  a  model  for  predicting  trans¬ 
fer  of  training  (TOT)  value  of  a  training  device.  This  report 
compares  and  contrasts  in  a  succinct,  critical  manner  several 
ways  of  trying  to  determine  a  priori  the  value  of  training 
devices.  Second,  it  presents  a  detailed  summary  of  the  effects 
of  various  conditions  and  variables  on  transfer  of  training. 
Within  the  scope  of  the  report,  both  topics  were  covered  well  . 
Two  tables  cross-referencing  transfer  variables  with  individual 
research  reports  alone  make  this  report  a  valuable  reference. 
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